Building a Three-Node Kubernetes Homelab That Fails Gracefully

A useful homelab is easy to start and surprisingly hard to operate. The first container is fun; the twentieth dependency is where design begins to matter. I rebuilt mine around a simple requirement: losing one ordinary component should create a degraded system, not an archaeological expedition.

Start with failure domains

The platform runs on three k3s nodes. Each participates in the control plane and embedded etcd, so cluster coordination does not depend on one machine. Critical workloads use anti-affinity to spread replicas, and PodDisruptionBudgets define how much capacity must remain during maintenance.

Three machines do not automatically create high availability. A service with three replicas can still fail if all replicas land on one node, share one database, or depend on one unmonitored network path. I treat availability as a chain: scheduling, ingress, state, name resolution, authentication, and recovery all need an explicit design.

Separate desired state from mutable state

Git contains the desired platform: manifests, generators, dashboards, alerts, scripts, and runbooks. It intentionally does not contain credentials, databases, uploaded files, private keys, or media.

Argo CD continuously reconciles the enabled production tree. First-party applications are built by GitHub Actions, pushed as commit-addressed images, and promoted into production by immutable digest. That creates a traceable path from a source commit to a container digest to the running Kubernetes revision.

Mutable application data lives on the storage tier that matches its recovery requirements. Longhorn replicates important volumes across nodes. CloudNativePG provides database-level replication. Bulk files use separate shared storage, and backups are verified independently. Replication helps availability; it does not replace a backup.

Keep the edge redundant

Two Caddy replicas terminate public TLS and route traffic into the cluster. Their readiness checks include real TLS handshakes for configured public names, which prevents a pod from receiving HTTPS traffic before it can serve the correct certificate.

Private services use separate addresses and access policies. Some are LAN or VPN only. Others use two-factor authentication through Authelia. A particularly sensitive mobile path requires a client certificate before the application login is even reachable. The point is not to use every security mechanism everywhere. It is to make the trust decision explicit for each route.

Preserve a rollback boundary

During the migration, I kept the original Docker host outside Kubernetes. It remains an administration and rollback machine instead of becoming another cluster node. That choice costs some hardware efficiency, but it avoids turning the system used to repair the cluster into part of the failure.

Design the operating system around the applications

The most important lesson is that Kubernetes is only one layer. The platform became dependable when deployment, access, DNS, data, observability, backup, and documentation were treated as one product.

Every feature must appear on three operational surfaces: Homepage for service discovery, an independently deployed Dashy catalog for directory redundancy, and Grafana for evidence. Contract tests assert those representations together. A workload is not finished just because its pod is running.