Model Router
A model routing service that distributes AI inference across local GPU
nodes with cloud fallback: LiteLLM Proxy on beast.lan routes
requests to llama.cpp servers on beast (AMD MI25) and athena
(RX5500 XT), with OpenRouter as the fallback. Open WebUI provides the
browser interface.
Status
- Status: active.
- Router endpoint:
beast.lan:4000/v1(OpenAI-compatible). - llama.cpp:
athena.lan:2001(RX5500 XT) andbeast.lan:2001(MI25). - Open WebUI:
beast.lan:8080. - Live endpoint state: see the dashboard.
How it works
Clients send OpenAI-compatible requests to the router
(beast.lan:4000/v1). LiteLLM evaluates the requested model and
routing rules, then dispatches to the selected llama.cpp server
(mi25-llama / rx5500-llama) or to OpenRouter
(openrouter-gpt-4o) as fallback, and returns the response
through the common endpoint.
Documentation
- Routing & execution map:
documents/05-model-router-map.md - Configuration:
sources/config/config.yaml(LiteLLM),docker-compose.yml - Deployment:
scripts/router-deploy.sh,sources/ansible/