feat(backup): hourly hot backup with a gated billing dump #517

Sammanfogat

supernaut sammanfogade 4 incheckningar från feat/158-hourly-backup in i main

2026-10-02 12:31:12 +00:00

supernaut kommenterad

2026-10-02 12:10:19 +00:00

Ägare

What

Rollout PRs 2 and 4 of #158 (ADR 0041), plus the runbook targets (PR 1's infra half).

Hourly hot backup (bitborg-backup-hot.sh, timer at :17): canary row in bitborg_backup.canary (portal DB, own schema), pg_dump -Z0 of the Forgejo and portal DBs, a Kanidm backup to its own file, then restic backup of the dumps and the Forgejo data volume into restic/hot on the primary off-site bucket. Forget --keep-hourly 48 --keep-daily 14 each run, --prune once a day. Excludes regenerable queues, indexers, tmp and push quarantine; restic exit 3 (file vanished mid-scan) is a warning with the snapshot kept.
Billing dump every 15 min, installed but stopped and disabled until billing_enabled.
Host safety: --host {{ inventory_hostname }}; backup_hot_enabled is true only on the prod host, so a scratch or restore host never writes into the prod repo. The units only After= Forgejo and Postgres, never start them.
Alerts: BackupHotStale (critical, over 3 h, also fires when the metric is absent), BackupHotFailed (status != 0 for 5 min), BackupHotBillingStale (over 1 h, rendered only when billing is enabled). Textfiles of disabled units are removed.
Runbook: recovery objectives per tier, and a new "Hourly hot backup (#158)" section: what it does, first run by hand, restore inside podman run, stop the hot timers before any restore, fall back to the previous snapshot if a live-read repo is inconsistent.

No new vault values: it reuses the primary off-site keys and the restic password.

Review

An independent review found 8 must-fix and 9 smaller issues; all fixed in 8131d60.

Verification

ansible-lint 0/0, --syntax-check OK, both scripts render and pass bash -n and shellcheck (local 0.11) with billing on and off, check-alert-rules 53/54 rules, check-metric-names OK. Not exercised on a host: the repositories-dir check and the exclude globs. The first manual run tests both.

Apply

Tags backup,monitoring. BackupHotStale fires right after the apply until the first success, so start the first full upload by hand as the runbook says. First upload is the whole Forgejo volume (several GB). No Forgejo or portal restart.

Refs #158

## What Rollout PRs 2 and 4 of #158 (ADR 0041), plus the runbook targets (PR 1's infra half). - **Hourly hot backup** (`bitborg-backup-hot.sh`, timer at :17): canary row in `bitborg_backup.canary` (portal DB, own schema), `pg_dump -Z0` of the Forgejo and portal DBs, a Kanidm backup to its own file, then `restic backup` of the dumps and the Forgejo data volume into `restic/hot` on the primary off-site bucket. Forget `--keep-hourly 48 --keep-daily 14` each run, `--prune` once a day. Excludes regenerable queues, indexers, tmp and push quarantine; restic exit 3 (file vanished mid-scan) is a warning with the snapshot kept. - **Billing dump every 15 min**, installed but stopped and disabled until `billing_enabled`. - **Host safety:** `--host {{ inventory_hostname }}`; `backup_hot_enabled` is true only on the prod host, so a scratch or restore host never writes into the prod repo. The units only `After=` Forgejo and Postgres, never start them. - **Alerts:** `BackupHotStale` (critical, over 3 h, also fires when the metric is absent), `BackupHotFailed` (status != 0 for 5 min), `BackupHotBillingStale` (over 1 h, rendered only when billing is enabled). Textfiles of disabled units are removed. - **Runbook:** recovery objectives per tier, and a new "Hourly hot backup (#158)" section: what it does, first run by hand, restore inside `podman run`, stop the hot timers before any restore, fall back to the previous snapshot if a live-read repo is inconsistent. No new vault values: it reuses the primary off-site keys and the restic password. ## Review An independent review found 8 must-fix and 9 smaller issues; all fixed in `8131d60`. ## Verification ansible-lint 0/0, `--syntax-check` OK, both scripts render and pass `bash -n` and shellcheck (local 0.11) with billing on and off, `check-alert-rules` 53/54 rules, `check-metric-names` OK. Not exercised on a host: the repositories-dir check and the exclude globs. The first manual run tests both. ## Apply Tags `backup,monitoring`. `BackupHotStale` fires right after the apply until the first success, so start the first full upload by hand as the runbook says. First upload is the whole Forgejo volume (several GB). No Forgejo or portal restart. Refs #158

supernaut lade till 4 incheckningar

2026-10-02 12:10:19 +00:00

docs(runbook): recovery objectives per tier bb7ad0b725