feat(backup-drill): restore and verify kanidm on the scratch vm #506

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/drill-kanidm-restore in i main

2026-09-30 17:59:31 +00:00

supernaut kommenterad

2026-09-30 17:55:13 +00:00

Ägare

What

The weekly backup drill now restores the Kanidm backup on the scratch VM and checks it.

Decompresses kanidm-backup.json (restore reads only plain JSON) and runs a one-shot kanidmd database restore with the pinned production image.
Re-exports the restored DB and counts persons and groups. Built-in, recycled and tombstoned entries are excluded.
Fails the drill if either count is zero, or if the restored counts differ from the counts in the dump.
Starts the restored server and probes /status.
Emits bitborg_backup_drill_kanidm_persons and bitborg_backup_drill_kanidm_groups. A failed check fails the drill, so BackupDrillFailed covers it. No new alert.

No admin credential is needed. The image and domain are templated from kanidm_image, kanidm_image_tag and kanidm_domain.

Verified

Count logic and restore round trip against a local Kanidm 1.11.1 DB: one person, one group and one deleted person gave persons=1, groups=1 in the dump and after restore.
Pre-push hooks pass (ansible-lint, ruff, gitleaks, markdownlint, prettier, metric names). shellcheck ran on the local 0.11 binary, not the pinned 0.10.
check-alert-rules.py, smoke-textfile.py, bash -n and py_compile on the rendered templates.

Not verified

The full drill has not run. After merge:

Apply with scripts/apply-reconcile.sh, tags backup-drill,monitoring.
Trigger bitborg-backup-drill.service.
In Loki: [restore] METRIC kanidm_persons, [restore] METRIC kanidm_groups, kanidm /status OK, then [drill] restore-drill PASSED.
Check both metrics are at least 1 and bitborg_backup_drill_last_run_status is 0.

Closes #458

## What The weekly backup drill now restores the Kanidm backup on the scratch VM and checks it. - Decompresses `kanidm-backup.json` (restore reads only plain JSON) and runs a one-shot `kanidmd database restore` with the pinned production image. - Re-exports the restored DB and counts persons and groups. Built-in, recycled and tombstoned entries are excluded. - Fails the drill if either count is zero, or if the restored counts differ from the counts in the dump. - Starts the restored server and probes `/status`. - Emits `bitborg_backup_drill_kanidm_persons` and `bitborg_backup_drill_kanidm_groups`. A failed check fails the drill, so `BackupDrillFailed` covers it. No new alert. No admin credential is needed. The image and domain are templated from `kanidm_image`, `kanidm_image_tag` and `kanidm_domain`. ## Verified - Count logic and restore round trip against a local Kanidm 1.11.1 DB: one person, one group and one deleted person gave persons=1, groups=1 in the dump and after restore. - Pre-push hooks pass (ansible-lint, ruff, gitleaks, markdownlint, prettier, metric names). shellcheck ran on the local 0.11 binary, not the pinned 0.10. - `check-alert-rules.py`, `smoke-textfile.py`, `bash -n` and `py_compile` on the rendered templates. ## Not verified The full drill has not run. After merge: 1. Apply with `scripts/apply-reconcile.sh`, tags `backup-drill,monitoring`. 2. Trigger `bitborg-backup-drill.service`. 3. In Loki: `[restore] METRIC kanidm_persons`, `[restore] METRIC kanidm_groups`, `kanidm /status OK`, then `[drill] restore-drill PASSED`. 4. Check both metrics are at least 1 and `bitborg_backup_drill_last_run_status` is 0. Closes #458

supernaut lade till 2 incheckningar

2026-09-30 17:55:13 +00:00

feat(backup-drill): restore kanidm backup and assert persons and groups 3e58f35288

Restores kanidm-backup.json with a one-shot container of the pinned prod image, re-exports it to count persons and groups without an admin credential, fails the drill on a zero count and probes /status on the restarted server. Counts are emitted as textfile metrics. Closes #458

fix(backup-drill): count only user groups and persons in kanidm restore check

ci / ci (pull_request) Successful in 1m45s

Detaljer

55ddb3eb8e

Built-in groups made the group count always pass. Skip builtin, recycled and tombstone entries, and fail when the restored counts differ from the dump. Also harden the metric write and keep stale kanidm metrics from surviving a run.

supernaut sammanfogade incheckning 302c176d35 till main

2026-09-30 17:59:31 +00:00

supernaut tog bort grenen feat/drill-kanidm-restore

2026-09-30 17:59:31 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-09-30 17:59:32 +00:00

feat(backup-drill): restore and verify kanidm on the scratch vm (#506)

supernaut refererade denna ändringsförfrågan från en incheckning

2026-10-02 17:56:58 +00:00

feat(backup-drill): restore and verify kanidm on the scratch vm (#506)

Logga in för att delta i denna konversation.

Inga granskare