Observability as a Product Surface, Not a Wall of Charts

Monitoring tells you that a number moved. Observability should help you decide what to do next.

That distinction shaped the Server Command Center in my homelab. It now contains 151 panels covering nodes, thermals, disks, storage, networking, Kubernetes, and incident-cause signals. The number sounds impressive, but panel count is not the goal. The goal is a short path from “this page does not load” to “this dependency is responsible.”

Begin at the user-visible route

Each important HTTP service has a synthetic probe that follows the path a user depends on. Mail adds protocol-aware checks for HTTPS, SMTP, submission, IMAP, POP3, and ManageSieve, including certificate validation where appropriate.

Those probes answer the first question: is the service usable from this vantage point? They do not explain the cause. For that, I correlate them with deployment readiness, pod restarts, node resources, volume state, database metrics, DNS resolution, certificate lifetime, and edge health.

An external scheduled check also tests public routes from outside the home network. Internal green lights are weak evidence when the ISP path, router, or public DNS is the failure.

Organize dashboards around decisions

The command center is broad because it is an infrastructure overview. Application dashboards are narrower. A scheduling service needs booking-route availability, database health, mail delivery signals, and integration status. A DNS dashboard needs resolver success, upstream latency, source distribution, and failure rates.

I avoid placing every metric on every dashboard. A panel earns space when it helps answer a concrete operational question.

Make alerts actionable

An alert without context transfers investigative work to the worst possible moment. My alert rules include a human-readable description and a runbook link. Queries are bounded so a broken exporter or unbounded label does not create its own incident.

I also separate immediate failures from capacity trends. A route that is down now needs a fast notification. A disk projected to fill in weeks needs a different threshold, cadence, and response.

Require three surfaces for every feature

I use Homepage as the canonical service-link directory, Dashy as an independent fallback directory, and Grafana as the evidence surface. A generator mirrors the Homepage catalog into Dashy, but Dashy runs as a separate application with separate replicas. Grafana remains independent of both.

Every new or changed feature must update all three representations, and its contract test asserts them together. A background automation without a user interface still gets a link to its dashboard, Argo CD application, or runbook.

This policy prevents the common failure where an application exists but nobody can find it, or a link exists but nobody can tell whether the application is healthy.

Treat dashboards as code

Dashboards, alert rules, scrape configuration, and probes live in Git. Generated artifacts are reproducible, and repository validation catches missing coverage before deployment. The same review that asks “does the application work?” can ask “how will we know?”