Designing a Local-First AI Platform Around an RTX Workstation
Local AI is often demonstrated as a model running in one terminal. A useful platform needs more: a stable interface, model policy, hardware-aware placement, speech services, authentication, recovery, and enough telemetry to explain why a request is slow.
I built Jarvis by separating the control plane from inference. Open WebUI runs in Kubernetes, while Ollama runs natively on a Windows workstation with an RTX 5080. The cluster provides the persistent, observable web application; the workstation provides the hardware-specific GPU runtime.
Do not force every workload into Kubernetes
The GPU already lived in a daily-use workstation. Passing it through to a cluster node would have complicated drivers, reboots, and desktop use. Running Ollama natively keeps the vendor GPU stack close to the hardware while exposing only a private API to the cluster.
The Kubernetes side handles authentication, routing, configuration, health checks, and the user interface. If the workstation is unavailable, the failure is visible and bounded instead of taking cluster control services with it.
Route by task, not model popularity
The model catalog includes different sizes and strengths. A custom routing pipe first classifies the request, then selects a larger coding model or a general-purpose model. The router is intentionally smaller and cheaper than the model doing the final work.
Fallback behavior matters. A routing policy must account for a model that is missing, loading, or too slow. The interface should report a real availability problem instead of silently returning a lower-quality answer that appears authoritative.
I benchmarked models on representative coding and general tasks rather than assuming parameter count predicts usefulness. The resulting catalog is managed as desired state.
Reuse services across interfaces
A CPU-based faster-whisper service provides private speech-to-text. Open WebUI, a standalone transcription lab, a browser voice console, and the local agent workstation all use the same API.
The model cache is treated as replaceable. Uploaded media and transcripts remain ephemeral unless the user deliberately saves them. That distinction simplifies backup policy and reduces the amount of sensitive audio retained after a task.
Image generation follows the same pattern. Fooocus runs natively on the GPU workstation, bound to the private network, while Caddy and two-factor authentication protect the browser route. The boot-time task, model path, and GPU compatibility check are documented and tested.
Observe AI as infrastructure
Model count, endpoint reachability, request latency, loading behavior, and GPU-service availability belong on operational dashboards. AI quality needs separate evaluation, but infrastructure failures should not masquerade as model failures.
The result is a platform, not a demo: private inputs stay local, interfaces share reusable services, hardware-specific workloads run where they fit, and failures are visible enough to repair.