Restore drill fails (BackupDrillFailed): weekly drill overlaps the daily storage restic backup's repo lock #219

Nytt ärende

Stängd

öppnade 2026-07-26 20:54:04 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-07-26 20:54:04 +00:00

Ägare

Symptom

BackupDrillFailed fires (critical); gitborg_backup_drill_last_run_status=1. The weekly restore drill (bitborg-backup-drill.service, Sat 05:00) fails at the [storage] restic restore with exit 11.

Root cause — restic repo lock collision

The daily storage-tier restic backup (bitborg-backup-storage.service, backup_restic_on_calendar = *-*-* 05:00:00) and the weekly restore drill (Sat *-*-* 05:00:00) run in the same slot and overlap. The backup holds restic's exclusive repository lock for its ~3-minute run; the drill's restic check tries to acquire the exclusive lock with no lock-wait and fails immediately:

unable to create lock in backend: repository is already locked by PID 1 ... on cb0691dabb50 by root
lock was created at 2026-07-25 05:02:02

Confirmed via Loki (2026-07-25): storage backup 05:01:58 → 05:04:58 (snapshot 01ca3b8f saved cleanly, 9.956 GiB); drill started 05:03:04; drill restic check at 05:04:02 → exit 11.

New failure mode since the storage-tier restic decoupling (#188 / #211): before that, [storage] lived in the age archive with no restic lock, so drill + backup never contended. Backups and the restore path are otherwise healthy — the backup snapshot is good and the drill restored Postgres + the data volume fine before hitting the lock. This is purely a lock-timing collision, not data loss.

Fix

Stagger the drill past the storage backup: backup_drill_on_calendar Sat 05:00 → Sat 06:00 (the daily storage backup at 05:00 finishes in ~3 min, so 06:00 is well clear).
Make the drill lock-resilient: add --retry-lock=10m to the drill's restic invocations (roles/backup-drill/templates/restore-on-scratch.sh.j2) so a concurrent backup is waited out rather than fatal (defence-in-depth).

After merge/apply, run one off-peak manual drill to confirm green and clear the firing alert.

## Symptom `BackupDrillFailed` fires (critical); `gitborg_backup_drill_last_run_status=1`. The weekly restore drill (`bitborg-backup-drill.service`, Sat 05:00) fails at the `[storage]` restic restore with **exit 11**. ## Root cause — restic repo lock collision The **daily** storage-tier restic backup (`bitborg-backup-storage.service`, `backup_restic_on_calendar = *-*-* 05:00:00`) and the **weekly** restore drill (`Sat *-*-* 05:00:00`) run in the same slot and overlap. The backup holds restic's **exclusive repository lock** for its ~3-minute run; the drill's `restic check` tries to acquire the exclusive lock with no lock-wait and fails immediately: ``` unable to create lock in backend: repository is already locked by PID 1 ... on cb0691dabb50 by root lock was created at 2026-07-25 05:02:02 ``` Confirmed via Loki (2026-07-25): storage backup 05:01:58 → 05:04:58 (snapshot `01ca3b8f` saved cleanly, 9.956 GiB); drill started 05:03:04; drill `restic check` at 05:04:02 → exit 11. New failure mode since the storage-tier restic decoupling (#188 / #211): before that, `[storage]` lived in the age archive with no restic lock, so drill + backup never contended. **Backups and the restore path are otherwise healthy** — the backup snapshot is good and the drill restored Postgres + the data volume fine before hitting the lock. This is purely a lock-timing collision, not data loss. ## Fix 1. **Stagger the drill past the storage backup**: `backup_drill_on_calendar` `Sat 05:00` → **`Sat 06:00`** (the daily storage backup at 05:00 finishes in ~3 min, so 06:00 is well clear). 2. **Make the drill lock-resilient**: add `--retry-lock=10m` to the drill's restic invocations (`roles/backup-drill/templates/restore-on-scratch.sh.j2`) so a concurrent backup is waited out rather than fatal (defence-in-depth). After merge/apply, run one off-peak manual drill to confirm green and clear the firing alert.

supernaut lade till etiketterna

area/backups

area/infra

ready-for-implementation

type

bug

2026-07-26 20:54:44 +00:00

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-07-26 20:55:12 +00:00

fix(backup-drill): avoid restic repo-lock collision with the storage backup (#219) #220

supernaut refererade till detta ärende från en incheckning

2026-07-26 20:55:12 +00:00