fix(backup-drill): report orphan repo dirs instead of failing fsck #525

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/158-drill-fsck-orphans in i main

2026-10-02 19:01:04 +00:00

supernaut kommenterad

2026-10-02 18:57:12 +00:00

Ägare

What

The second production drill run (with #524's logging) got through the hot leg's restore, pg_restore and canary check (snapshot and canary 9 s apart). It failed only on git fsck: 15 repos checked, 1 failed with "not a git repository". That directory, supernaut/migration-verify-scratch.git, is an orphan in production too. It holds only an empty pulls/ subdir, no HEAD and no objects, and its repo no longer exists. The backup copied it correctly.

*.git dirs without HEAD and objects are logged as ORPHAN <path>, counted, and skipped. They no longer fail the leg.
A real repo that fails fsck, or zero real repos, still fails the leg.
FSCK_DONE <checked> <bad> <orphans>, plus the new metric bitborg_backup_drill_hot_orphan_dirs.
Runbook: orphans are reported, not failed; find and remove them under Site Administration, Unadopted Repositories.

Verification

Local fixture: three bare repos (one corrupted) and one orphan gives FSCK_DONE 3 1 1, rc=1; without the corrupted repo, FSCK_DONE 2 0 1, rc=0. --syntax-check, ansible-lint 0, bash -n, py_compile, check-metric-names, prettier and markdownlint pass.

Apply

Tag backup-drill: only the on-VM hot restore script and the orchestrator change. Then run the drill by hand; both legs should pass.

Refs #158

## What The second production drill run (with #524's logging) got through the hot leg's restore, `pg_restore` and canary check (snapshot and canary 9 s apart). It failed only on `git fsck`: 15 repos checked, 1 failed with "not a git repository". That directory, `supernaut/migration-verify-scratch.git`, is an orphan in production too. It holds only an empty `pulls/` subdir, no `HEAD` and no objects, and its repo no longer exists. The backup copied it correctly. - `*.git` dirs without `HEAD` and `objects` are logged as `ORPHAN <path>`, counted, and skipped. They no longer fail the leg. - A real repo that fails fsck, or zero real repos, still fails the leg. - `FSCK_DONE <checked> <bad> <orphans>`, plus the new metric `bitborg_backup_drill_hot_orphan_dirs`. - Runbook: orphans are reported, not failed; find and remove them under Site Administration, Unadopted Repositories. ## Verification Local fixture: three bare repos (one corrupted) and one orphan gives `FSCK_DONE 3 1 1`, rc=1; without the corrupted repo, `FSCK_DONE 2 0 1`, rc=0. `--syntax-check`, ansible-lint 0, `bash -n`, `py_compile`, check-metric-names, prettier and markdownlint pass. ## Apply Tag `backup-drill`: only the on-VM hot restore script and the orchestrator change. Then run the drill by hand; both legs should pass. Refs #158

supernaut lade till 1 incheckning

2026-10-02 18:57:12 +00:00

fix(backup-drill): report orphan repo dirs instead of failing fsck

ci / ci (pull_request) Successful in 1m47s

Detaljer

b17ff7c9f2