The challenge
What the system needed to solve
Offer useful code and general-purpose AI without sending private repositories, prompts, or operational context to third-party inference services.
The build
Engineering decisions
01
Connected a Kubernetes-hosted Open WebUI control plane to Ollama running natively on a Windows RTX 5080 workstation.
02
Created a quality-first routing pipe that classifies requests and selects between larger coding and general-purpose models.
03
Added startup recovery, reachability checks, model inventory management, latency telemetry, and explicit local-only policy documentation.
04
Benchmarked candidate models on representative prompts instead of selecting them solely by parameter count.
Architecture
How the pieces connect
- 1Authenticated browser or tool
- 2Open WebUI and routing pipe
- 3Private network API
- 4Ollama on RTX workstation
- 5Local model storage
Results
What changed
- Sensitive coding and homelab operations remain within the private network.
- Users get a single interface while routing policy chooses the appropriate local model.
- Model availability and performance are observable as infrastructure, not treated as a black box.