The challenge
What the system needed to solve
Protect mutable data without pretending that replication is a backup or filling Git with databases, secrets, and media.
The build
Engineering decisions
Classified data by recovery need: Git-rebuildable configuration, replicated cluster state, bulk NAS data, encrypted backup sets, and disposable caches.
Used Longhorn for replicated application volumes and CloudNativePG for database high availability.
Built capacity, inode, mount-state, backup-age, and time-to-full monitoring with alerts before failures become emergencies.
Documented bare-metal rebuilds, cluster recovery order, application restore checks, and evidence required before retiring old copies.
Architecture
How the pieces connect
- 1Application state
- 2Replicated Longhorn or PostgreSQL
- 3Encrypted backup jobs
- 4Separate bulk target
- 5Restore verification
Results
What changed
- Recovery procedures identify what can be rebuilt, what must be restored, and what must never enter source control.
- Storage growth and backup freshness are visible before capacity or retention becomes critical.
- The original host and cold artifacts remain available until replacement paths pass real restore checks.
