fix(backup): harden retention + off-site failure handling (M6/M7/L2/L3) #147

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/126-backup-robustness in i main

2026-07-19 17:52:23 +00:00

supernaut kommenterad

2026-07-19 17:39:58 +00:00

Ägare

Scope (#126 — genuine data-safety bugs)

The bug subset of the backup-robustness audit. Design items (H6 DR-key escrow, H7 PITR/WAL, M5 hot-skew, M9 multipart abort, M10 full RTO) are not in this PR — they'll be tracked as separate follow-ups. All changes are in roles/backup/templates/bitborg-backup.sh.j2.

M6 — orphaned `*.tar.age.partial` leak

A SIGKILL/OOM mid-age skips the EXIT trap, leaving a multi-GB bitborg-*.tar.age.partial that pruning (which matches *.tar.age, not *.partial) never removes → compounds disk-fill run after run. Fix: sweep stale partials at run start (any present then are from an aborted prior run — this run's ENC name is timestamped and not created yet).

M7 — off-site failure escalated to "total failure" + halted local prune

An off-site upload failure exit 1'd before the local prune and before recording local success, so a Glesys outage both masked a good local backup (run went red) and halted local pruning (disk fill). Fix: off-site failure no longer fails the run or skips the local prune — it's a WARNING; local success is recorded independently, and the per-destination BackupOffsiteFailed alert (already wired) surfaces the off-site problem.

L2 — no keep-last-N floor

Local prune was time-only, so >14 days of failed/degraded backups would prune every good copy. Fix: backup_keep_min (default 3) — always retain the newest N regardless of age. Functionally tested: with all archives older than retention, the newest 3 still survive.

L3 — off-site prune had no name filter

rclone delete --min-age matched any object in the bucket. Fix: --include "bitborg-*.tar.age" --include "bitborg-*.tar.enc" so it only ever deletes our own archives.

Validation

ansible-lint (production profile) · --syntax-check · --check --diff renders clean (0 failed) · bash -n on the rendered script · functional test of the keep-N prune (normal + all-expired edge case).

Apply notes

--tags backup. Re-renders the script only; effect is on the next scheduled/manual backup run. New knob: backup_keep_min in group_vars/all/vars.yml.

Refs #126.

## Scope (#126 — genuine data-safety bugs) The bug subset of the backup-robustness audit. Design items (H6 DR-key escrow, H7 PITR/WAL, M5 hot-skew, M9 multipart abort, M10 full RTO) are **not** in this PR — they'll be tracked as separate follow-ups. All changes are in `roles/backup/templates/bitborg-backup.sh.j2`. ### M6 — orphaned `*.tar.age.partial` leak A SIGKILL/OOM mid-`age` skips the EXIT trap, leaving a multi-GB `bitborg-*.tar.age.partial` that pruning (which matches `*.tar.age`, not `*.partial`) never removes → compounds disk-fill run after run. **Fix:** sweep stale partials at run start (any present then are from an aborted prior run — this run's ENC name is timestamped and not created yet). ### M7 — off-site failure escalated to "total failure" + halted local prune An off-site upload failure `exit 1`'d before the local prune and before recording local success, so a Glesys outage both **masked a good local backup** (run went red) and **halted local pruning** (disk fill). **Fix:** off-site failure no longer fails the run or skips the local prune — it's a WARNING; local success is recorded independently, and the per-destination `BackupOffsiteFailed` alert (already wired) surfaces the off-site problem. ### L2 — no keep-last-N floor Local prune was time-only, so >14 days of failed/degraded backups would prune **every** good copy. **Fix:** `backup_keep_min` (default 3) — always retain the newest N regardless of age. Functionally tested: with all archives older than retention, the newest 3 still survive. ### L3 — off-site prune had no name filter `rclone delete --min-age` matched **any** object in the bucket. **Fix:** `--include "bitborg-*.tar.age" --include "bitborg-*.tar.enc"` so it only ever deletes our own archives. ## Validation ansible-lint (production profile) · `--syntax-check` · `--check --diff` renders clean (0 failed) · `bash -n` on the rendered script · functional test of the keep-N prune (normal + all-expired edge case). ## Apply notes `--tags backup`. Re-renders the script only; effect is on the next scheduled/manual backup run. New knob: `backup_keep_min` in `group_vars/all/vars.yml`. Refs #126.

supernaut lade till 1 incheckning

2026-07-19 17:39:58 +00:00

fix(backup): harden retention + off-site failure handling

ci / ci (pull_request) Successful in 2m16s

Detaljer

69d6ed4d41

Four data-safety bug fixes to gitborg-backup.sh.j2 from the
pre-onboarding audit (#126):

- M6: sweep orphaned gitborg-*.tar.age.partial at run start (a SIGKILL/
  OOM mid-age skips the EXIT trap; pruning never removes partials).
- M7: an off-site upload failure no longer fails the whole run or skips
  the local prune — it previously masked a good local backup and halted
  pruning (disk fill). Off-site is surfaced via BackupOffsiteFailed;
  local success is recorded independently.
- L2: keep-last-N floor (backup_keep_min, default 3) so a run of failed/
  degraded backups longer than retention can't prune every good copy.
- L3: name-filter the off-site prune (--include gitborg-*.tar.age/.enc)
  so it only ever deletes our own archives.

Validated: ansible-lint (production), --check renders clean, bash -n on
the rendered script, functional test of the keep-N prune.

Refs #126.

supernaut sammanfogade incheckning c931b73f9f till main

2026-07-19 17:52:23 +00:00

supernaut tog bort grenen fix/126-backup-robustness

2026-07-19 17:52:23 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-19 17:52:24 +00:00

fix(backup): harden retention + off-site failure handling (M6/M7/L2/L3) (#147)

supernaut refererade till denna ändringsförfrågan

2026-07-19 18:47:44 +00:00

Backup robustness for production (DR key drill, RPO/PITR, hot-skew, partial leak, off-site failure, RTO) #126

supernaut refererade till denna ändringsförfrågan

2026-07-19 19:01:26 +00:00