The challenge
What the system needed to solve
Answer not only whether a service is down, but why: node pressure, storage saturation, DNS failure, certificate expiry, networking, or a bad rollout.
The build
Engineering decisions
Created a 151-panel Grafana command center covering node resources, thermals, disks, SMART data, storage, networking, Kubernetes, and incident-cause signals.
Combined Prometheus metrics with protocol-aware Blackbox probes, kube-state-metrics, exporters, and Uptime Kuma checks.
Designed actionable alerts with bounded queries, human-readable annotations, and runbook links.
Established a three-surface contract: every feature appears in Homepage, an independently generated Dashy catalog, and Grafana evidence.
Architecture
How the pieces connect
- 1Applications, nodes, and external routes
- 2Exporters and synthetic probes
- 3Prometheus
- 4Grafana and Alertmanager
- 5Telegram incident delivery
Results
What changed
- Failures can be traced from a user-facing route to the responsible service, node, volume, or network dependency.
- Dashboard and alert definitions are generated and contract-tested in Git.
- Off-site checks provide independent evidence when the home network itself is unavailable.
