Model Routing & Execution Map

Harvested 2026-08-24 from the failed ~/work/model-router-setup attempt (docs/model-routing-map.md, prompts/01-goals.md, prompts/01-topology.md). Live-state facts here need verification; see TODO.

Architecture

• beast.lan — LiteLLM Proxy (port 4000) + PostgreSQL. The API gateway, router, and usage tracker.
• athena.lan — llama.cpp server (port 2001). Runs small fast utility models (tagging, summarization, classification, lightweight inference).
• beast.lan (optional local) — llama.cpp (port 2001) for heavy GGUF models on the MI25.
• OpenRouter — cloud fallback for frontier models, heavy reasoning, and overflow; accessed through LiteLLM fallback chains.

Hardware

• Beast: AMD MI25 GPU (16 GB VRAM), dual Xeon, 256 GB RAM. Runs heavy local GGUF models; llama.cpp compiled with ROCm; uses GPU VRAM plus system RAM. Settings: sources/config/beast-config.ini.
• Athena: AMD RX5500 XT (8 GB VRAM), Ryzen 9 5900X, 128 GB RAM. Runs small models with GPU offload and CPU/RAM fallback. Settings: sources/config/athena-config.ini.

Request Flow

1. Client sends an OpenAI-compatible request to beast.lan:4000/v1.
2. LiteLLM evaluates the requested model and routing rules.
3. The request goes to Beast, Athena, or OpenRouter according to the configured route.
4. The selected llama.cpp server performs local inference when a local backend is used.
5. LiteLLM returns the response through the common endpoint.

LiteLLM Routing Configuration (sources/config/config.yaml)

Known discrepancy (see TODO): both mi25-llama and rx5500-llama currently point at athena; the topology says the MI25 lives on beast. Reconcile when the local endpoints are verified.

GPU Execution

llama.cpp uses --gpu-layers for automatic dynamic offload:

• Small/medium models (1B–7B Q4/Q8) fit entirely in GPU VRAM.
• Large models (14B–32B) partially offload; remaining layers run on CPU/RAM.
• Oversized models (70B–120B) run mostly CPU/RAM with GPU acceleration for the layers that fit.

Network Layout

• beast.lan — LiteLLM 4000, Open WebUI 8080, PostgreSQL 5432, optional llama.cpp 2001.
• athena.lan — llama.cpp 2001.
• OpenRouter — external cloud API via beast’s network connection.

Supporting Services

• PostgreSQL on beast: LiteLLM usage tracking and virtual keys.
• Open WebUI on beast: browser interface at http://beast.lan:8080.
• Docker Compose (sources/config/docker-compose.yml): LiteLLM, PostgreSQL, Open WebUI as one stack.

Deployment

• Ansible inventory: sources/ansible/inventory.yml.
• Playbooks: sources/ansible/playbooks/.
• One-command deploy: sh scripts/router-deploy.sh (wraps ansible-playbook for athena then beast).

Adding a New Model

1. Ensure the GGUF is downloaded and accessible to llama.cpp on the target host.

2. Add an entry to sources/config/config.yaml under model_list:

   - model_name: my-new-model
         litellm_params:
           model: openai/<model-repo>:<quant>
           api_base: http://athena.lan:2001/v1
           api_key: dummy

3. Run sh scripts/router-deploy.sh to push the updated config and restart LiteLLM.