No restore drill, no backup

The most common answer to "do we have backups?" is "yes, it's installed". But having a backup tool installed doesn't mean backups are being taken. Setups with a green dashboard and a valid license but not a single backup policy defined are surprisingly common.

This post covers the path from "it's installed" to "we can delete a namespace and bring it back".

1. What can you lose?

In Kubernetes, "backup" isn't one thing. There are three separate loss scenarios, and each needs its own backup:

If databases live outside the cluster, they are a separate layer of their own (for PostgreSQL, point-in-time recovery (PITR) using a WAL archive).

2. Setup: three layers, one target

• Velero backs up all Kubernetes objects and triggers CSI snapshots of the volumes.
• The Longhorn backup target moves those snapshots to S3-compatible storage (MinIO). Velero's VolumeSnapshotClass uses Longhorn's backup type (type: bak).
• RKE2 etcd snapshots are written every 12 hours to both local disk and S3.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: velero-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"      # every night at 02:00
  template:
    ttl: 336h0m0s            # 14 days
    excludedNamespaces: [velero]
    snapshotMoveData: false

The backup target must be in a separate location. A disk in the same data center is useless once that data center is gone.

Least privilege

Each cluster's backup user can write only to its own folder in S3. If one cluster is compromised, it can't delete the other clusters' backups. Delete permission is granted only where old snapshots actually need to be cleaned up.

3. The real work: the restore drill

Once the setup is done, the backup list starts filling up and everyone relaxes. But the only proof that a backup works is being able to restore from it.

The drill should be simple and repeatable:

# 1. Test namespace: a small PVC with a random file in it
kubectl create ns velero-test
# ... PVC + pod, note down the file's md5

# 2. Take a backup
velero backup create test-1 --include-namespaces velero-test --wait

# 3. Delete the namespace
kubectl delete ns velero-test

# 4. Restore and compare the file's md5
velero restore create --from-backup test-1 --restore-volumes=true --wait

If the file's md5 matches, both layers (objects + volumes) are working together. Run this drill on every cluster separately; it working on one cluster tells you nothing about another.

4. What can a drill uncover?

Two examples you would never see without a drill:

• Velero memory limit: If a cluster accumulates a large number of custom resources (e.g. security scan reports), Velero can hit its default memory limit and get OOM-killed. The backup looks "installed", but that night's backup never happened. Always set up alerts on the outcome of backup jobs.
• Secrets needed for restore: Restoring from an etcd snapshot requires the RKE2 server token; restoring from an encrypted Rancher backup requires the encryption key. These must live outside the server, in a password vault. A key that is lost along with the server makes the backup useless too.

5. The "which backup for which need" table

When something breaks at 03:00, nobody wants to read documentation. Put a single table in the handover document:

Next to any row that hasn't been drilled yet, write "not drilled" explicitly. An honest table is worth more than an optimistic one.

Lesson

Installing a backup tool isn't even half the job. Policy, alerting, a separate location, keys in the vault and, above all, regular restore drills. A backup that has never been restored is just hope.