feat(backup-drill): restore and verify the hourly hot backup #521

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/158-drill-hot in i main

2026-10-02 16:33:30 +00:00

supernaut kommenterad

2026-10-02 16:14:47 +00:00

Ägare

What

Rollout PR 3 of #158 (ADR 0041): the weekly drill also proves the hourly hot backup restores.

Freshness: newest hot snapshot for this host must be at most 65 min older than the drill start.
Restore on the host (restic container, same env as the hot job) into a temp dir on the backup volume, then stream it to the scratch VM over ssh … tar. No restic or S3 credentials reach the VM. Free space checked first (backup_drill_hot_min_free_gb, 2); temp dir removed in finally.
On the VM: pg_restore the portal dump into scratch Postgres and assert bitborg_backup.canary.at is within 2 min of the snapshot time; git fsck --connectivity-only on every restored repo, fail on any error or zero repos.
Metrics: bitborg_backup_drill_hot_status, …_snapshot_age_seconds, …_repos_checked. A hot failure fails the overall drill, so BackupDrillFailed fires (description extended). Gated by backup_drill_hot_enabled (follows backup_hot_enabled).
Runbook drill section updated. Timeout unchanged at 3600 s.

Verification

--syntax-check, ansible-lint 0, rendered templates pass py_compile, bash -n, shellcheck (one intended SC2016 info). Local unit checks: timestamp parsing on four formats, the metric writer, and the fsck loop against good and corrupted bare repos. Not yet run end to end: the first manual drill after the apply is the test.

Closes #158

## What Rollout PR 3 of #158 (ADR 0041): the weekly drill also proves the hourly hot backup restores. - Freshness: newest `hot` snapshot for this host must be at most 65 min older than the drill start. - Restore on the host (restic container, same env as the hot job) into a temp dir on the backup volume, then stream it to the scratch VM over `ssh … tar`. No restic or S3 credentials reach the VM. Free space checked first (`backup_drill_hot_min_free_gb`, 2); temp dir removed in `finally`. - On the VM: `pg_restore` the portal dump into scratch Postgres and assert `bitborg_backup.canary.at` is within 2 min of the snapshot time; `git fsck --connectivity-only` on every restored repo, fail on any error or zero repos. - Metrics: `bitborg_backup_drill_hot_status`, `…_snapshot_age_seconds`, `…_repos_checked`. A hot failure fails the overall drill, so `BackupDrillFailed` fires (description extended). Gated by `backup_drill_hot_enabled` (follows `backup_hot_enabled`). - Runbook drill section updated. Timeout unchanged at 3600 s. ## Verification `--syntax-check`, ansible-lint 0, rendered templates pass `py_compile`, `bash -n`, shellcheck (one intended SC2016 info). Local unit checks: timestamp parsing on four formats, the metric writer, and the fsck loop against good and corrupted bare repos. Not yet run end to end: the first manual drill after the apply is the test. Closes #158

supernaut lade till 2 incheckningar

2026-10-02 16:14:47 +00:00

feat(backup-drill): restore and verify the hourly hot backup b2a9596254

The weekly drill now checks the newest hot snapshot is at most 65 min
old, restores it on the scratch VM, restores the portal DB and checks
the canary is within 2 min of the snapshot time, and runs
git fsck --connectivity-only on every restored repository. It exports
bitborg_backup_drill_hot_status, _hot_snapshot_age_seconds and
_hot_repos_checked. The leg is gated by backup_hot_enabled. The unit
timeout goes from 60 to 90 min for the second full-volume restore.

Closes #158

fix(backup-drill): restore the hot snapshot on the host, not the VM

ci / ci (pull_request) Successful in 1m51s

Detaljer

9c09ebd840

The restic password and S3 keys no longer reach the scratch VM. The host
restores the snapshot in a container into the drill work dir, checks the
backup volume has free space first, and streams it as a tar into the VM.
The host directory is removed in the finally block. The unit timeout goes
back to 3600 s: the snapshot is small.

Closes #158

supernaut tvångsskickade feat/158-drill-hot från 9c09ebd840

ci / ci (pull_request) Successful in 1m51s

Detaljer

till 2cf944b23c

ci / ci (pull_request) Successful in 1m57s

Detaljer

2026-10-02 16:31:03 +00:00

Jämför

supernaut schemalade den här ändringsförfrågan för automatisk sammanfogning när alla kontroller lyckas

2026-10-02 16:31:08 +00:00

supernaut sammanfogade incheckning 96b866c0ae till main

2026-10-02 16:33:30 +00:00