fix(backup-drill): avoid restic repo-lock collision with the storage backup (#219) #220

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/backup-drill-restic-lock in i main

2026-07-26 21:08:16 +00:00

supernaut kommenterad

2026-07-26 20:55:12 +00:00

Ägare

Fixes #219 — the weekly restore drill fails (BackupDrillFailed, exit 11) when it overlaps the daily storage-tier restic backup's exclusive repo lock.

Stagger: backup_drill_on_calendar Sat 05:00 → Sat 06:00 (the daily storage backup at 05:00 finishes in ~3 min).
Resilience: --retry-lock=10m on the drill's restic wrapper (restore-on-scratch.sh) so a concurrent backup is waited out, not fatal.

Root cause + Loki timeline in #219. Backups and the restore path are healthy — the backup snapshot saved cleanly and the drill restored Postgres + the data volume fine before hitting the lock; purely a scheduling/lock-timing collision introduced by the storage-tier restic decoupling (#188/#211).

After merge + apply, a manual off-peak drill will confirm green and clear the alert. ansible:check, ansible-lint (production), Prettier clean.

Closes #219.

Fixes #219 — the weekly restore drill fails (`BackupDrillFailed`, exit 11) when it overlaps the daily storage-tier restic backup's exclusive repo lock. - **Stagger:** `backup_drill_on_calendar` Sat 05:00 → Sat 06:00 (the daily storage backup at 05:00 finishes in ~3 min). - **Resilience:** `--retry-lock=10m` on the drill's restic wrapper (`restore-on-scratch.sh`) so a concurrent backup is waited out, not fatal. Root cause + Loki timeline in #219. Backups and the restore path are healthy — the backup snapshot saved cleanly and the drill restored Postgres + the data volume fine before hitting the lock; purely a scheduling/lock-timing collision introduced by the storage-tier restic decoupling (#188/#211). After merge + apply, a manual off-peak drill will confirm green and clear the alert. `ansible:check`, `ansible-lint` (production), Prettier clean. Closes #219.

supernaut lade till 1 incheckning

2026-07-26 20:55:13 +00:00

fix(backup-drill): avoid restic repo-lock collision with the storage backup (#219 )

ci / ci (pull_request) Successful in 1m24s

Detaljer

10cbbaab42

The weekly restore drill (Sat 05:00) overlapped the daily storage-tier restic
backup (05:00, ~3 min), whose exclusive repo lock made the drill's restic check
fail immediately with exit 11 -> BackupDrillFailed. Stagger the drill to Sat 06:00
(clear of the 05:00 backup) and add --retry-lock=10m to the drill's restic calls so
a concurrent backup is waited out rather than fatal. Backups + restore path are
healthy; this is purely a lock-timing fix.

supernaut sammanfogade incheckning 0575e4e39d till main

2026-07-26 21:08:16 +00:00

supernaut tog bort grenen fix/backup-drill-restic-lock

2026-07-26 21:08:16 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-26 21:08:16 +00:00

fix(backup-drill): avoid restic repo-lock collision with the storage backup (#219) (#220)

supernaut refererade till denna ändringsförfrågan

2026-07-26 21:14:05 +00:00

feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226) #221

supernaut refererade till denna ändringsförfrågan

2026-07-26 21:21:43 +00:00