Data safety: recovery that has been proven, not assumed

supernaut kommenterad

2026-09-24 13:04:40 +00:00

Ägare

Backups exist and a weekly drill restores the git host. That is not yet the same as proven recovery. The identity provider is never restored, the disaster-recovery key has never been used, there is one daily dump with no point-in-time recovery, one operator holds every key, and one cloud credential can delete both the service and its backups.

This epic makes recovery something that has been shown to work, before anyone's data depends on it.

Tasks

bitborg/bitborg-infra#458 — restore and verify the identity provider in the backup drill
bitborg/bitborg-infra#157 — decrypt-test the disaster-recovery key and keep a second copy
bitborg/bitborg-infra#158 — hourly hot backup, RPO 1 h (ADR 0041 chose hourly dumps over WAL archiving)
bitborg/bitborg-infra#465 — one OS user and one credential per service
bitborg/bitborg-infra#468 — second operator or key escrow
bitborg/bitborg-infra#470 — OpenTofu state off-site
bitborg/bitborg-infra#159 — backup consistency
bitborg/bitborg-infra#161 — measured recovery time
bitborg/bitborg-infra#529: retire the pre-rename bucket (2026-10-30) and archive prefix (2026-10-31)

Done when

A full restore of git data, the portal database and the identity provider has been run from off-site copies using only the escrowed key, and the time it took is recorded.

Horizon: Now.

Backups exist and a weekly drill restores the git host. That is not yet the same as proven recovery. The identity provider is never restored, the disaster-recovery key has never been used, there is one daily dump with no point-in-time recovery, one operator holds every key, and one cloud credential can delete both the service and its backups. This epic makes recovery something that has been shown to work, before anyone's data depends on it. ## Tasks - [x] bitborg/bitborg-infra#458 — restore and verify the identity provider in the backup drill - [ ] bitborg/bitborg-infra#157 — decrypt-test the disaster-recovery key and keep a second copy - [x] bitborg/bitborg-infra#158 — hourly hot backup, RPO 1 h (ADR 0041 chose hourly dumps over WAL archiving) - [ ] bitborg/bitborg-infra#465 — one OS user and one credential per service - [ ] bitborg/bitborg-infra#468 — second operator or key escrow - [ ] bitborg/bitborg-infra#470 — OpenTofu state off-site - [ ] bitborg/bitborg-infra#159 — backup consistency - [ ] bitborg/bitborg-infra#161 — measured recovery time - [ ] bitborg/bitborg-infra#529: retire the pre-rename bucket (2026-10-30) and archive prefix (2026-10-31) ## Done when A full restore of git data, the portal database and the identity provider has been run from off-site copies using only the escrowed key, and the time it took is recorded. **Horizon:** Now.

supernaut lade till etiketterna

2026-09-24 13:04:53 +00:00

supernaut lade till detta till projektet Bitborg Roadmap

2026-09-24 13:04:55 +00:00

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-09-24 13:04:55 +00:00

backup-drill: restore and verify Kanidm on the scratch VM #458

supernaut refererade till detta ärende från bitborg/bitborg-infra