LLM Integration — project status

This project owns the local-LLM integration story for the home network: which models run on which host, how every tool reaches them, and the policies for splitting work between GPU-fast and CPU-heavy models.

What is live

• Two llama.cpp routers (router mode, presets, aliases): the heavy server (GPU + dual-Xeon CPU) and the desktop server (GPU + CPU fallbacks).
• Tuned service configs: parallel 4, cache-reuse 256, KV-cache quantization (q8_0 common, q4_0 on the large-context gemma presets), an embeddings endpoint for retrieval work.
• DeepSeek Harness provider routes to both routers, with the model catalog, retention policy, and routing policy documented here.
• Tooling guides for aider, the editor extension, and the travel-laptop deployment plan (a laptop with a local Vulkan build of llama.cpp).

Status

• Status: complete (declared 2026-08-26).
• Phase 1 (initial integration and deployment) complete: routers tuned, catalog and retention settled, providers wired, travel plan deployed.
• Deferred: retrieval-augmented chat UI (decided not to pursue for now).

How it works

Facts are captured from the live routers at build time, never hand-copied: model lists, contexts, and aliases come from the captured endpoint responses in dataflow.in/endpoints/. Documents carry verified <date> markers for live-state facts. The published pages are built by scripts/site-pages.sh (the homelab project-pages conventions) and published by the homelab project.