bitborg/bitborg-infra
Bevaka
2
0
Avgrena
Du har redan avgrenat bitborg-infra
 0
Kod Ärenden 64 Ändringsförfrågningar 3 Projekt 1 Utgåvor Paket Wiki Aktiviteter Actions

Backups: CI/scratch-instance test-restore job #40

Nytt ärende
Stängd
öppnade 2026-07-09 23:03:57 +00:00 av supernaut · 1 kommentar
supernaut kommenterad 2026-07-09 23:03:57 +00:00
Ägare
Kopiera länk

A weekly on-host restore-verification timer already runs. Add the CI/scratch-instance variant: restore into a scratch instance and run forgejo doctor.

Epic: gitborg/gitborg-docs#2

A weekly on-host restore-verification timer already runs. Add the CI/scratch-instance variant: restore into a scratch instance and run `forgejo doctor`. Epic: gitborg/gitborg-docs#2
supernaut refererade till detta ärende från en incheckning 2026-07-11 20:17:38 +00:00
docs: add ADR 0027 (automated restore drill)
supernaut kommenterad 2026-07-11 20:19:00 +00:00
Upphovsperson
Ägare
Kopiera länk

Implemented, deployed, and verified live on prod. 🎉

The CI/scratch-instance variant now runs as a weekly bitborg-backup-drill systemd timer on the services host: it fetches the latest off-site archive, decrypts it with the on-host verify key, boots a throwaway OpenStack VM (ADR 0021 pattern, no FIP), restores Postgres + the Forgejo data volume into a clean stack, runs forgejo doctor, records a metric, and tears the VM down. Orchestration lives on the services host (not the CI runner pool) to keep the OpenStack credential + backup-decrypt capability least-privilege.

Verified end-to-end: a live run restored the latest off-site archive into a throwaway VM; forgejo doctor confirmed 5 repos + DB consistency + version 305; the VM was torn down with no leaked server/volume; gitborg_backup_drill_last_run_status = 0 is scraped into VictoriaMetrics; BackupDrillFailed / BackupDrillStale alerts are loaded. (It took 8 runs to go green — each surfaced a real integration bug, which is exactly the point of a restore drill.)

Sibling tasks #39 (Bahnhof S3 migration, blocked on the DPA) and #41 (second off-provider destination) remain open under epic gitborg/gitborg-docs#2.

Implemented, deployed, and **verified live on prod**. 🎉 The CI/scratch-instance variant now runs as a weekly `bitborg-backup-drill` systemd timer on the services host: it fetches the latest **off-site** archive, decrypts it with the on-host verify key, boots a throwaway OpenStack VM (ADR 0021 pattern, no FIP), restores Postgres + the Forgejo data volume into a clean stack, runs `forgejo doctor`, records a metric, and tears the VM down. Orchestration lives on the services host (not the CI runner pool) to keep the OpenStack credential + backup-decrypt capability least-privilege. **Verified end-to-end:** a live run restored the latest off-site archive into a throwaway VM; `forgejo doctor` confirmed **5 repos + DB consistency + version 305**; the VM was torn down with no leaked server/volume; `gitborg_backup_drill_last_run_status = 0` is scraped into VictoriaMetrics; `BackupDrillFailed` / `BackupDrillStale` alerts are loaded. (It took 8 runs to go green — each surfaced a real integration bug, which is exactly the point of a restore drill.) - Implementation + verification: gitborg/gitborg-infra#66 - Decision record (ADR 0027) + architecture: gitborg/gitborg-docs#17 Sibling tasks #39 (Bahnhof S3 migration, blocked on the DPA) and #41 (second off-provider destination) remain open under epic gitborg/gitborg-docs#2.
supernaut refererade till detta ärende från en incheckning 2026-07-11 20:39:30 +00:00
docs: add ADR 0027 (automated restore drill)
Logga in för att delta i denna konversation.
Ingen gren/etikett specificerad
Grenar Taggar
main
fix/log-retention-and-erasure
feat/billing-reconcile-trigger
chore/renovate-script-pins
Inga resultat hittades.
Etiketter
Rensa etiketter
  
area/backups
Backup & restore

  
area/ci
Actions / runners

  
area/control-panel
Logged-in self-service area

  
area/identity
Kanidm / OIDC / entitlements

  
area/infra
Infrastructure (OpenTofu/Ansible/host)

  
area/observability
Monitoring / logs / analytics

  
area/payments
Billing / payment providers

  
area/security
Security & compliance (incl. ISO 27001)

  
area/storage
Volumes / data / LFS

  
area/web
Public web portal (gitborg-web)

  
blocked
Evaluated and specified, but waiting on an external dependency

  
needs-info
Waiting on the reporter for more information

  
needs-triage
Not yet evaluated

  
ready-for-implementation
Fully specified, ready to be picked up

  
type
bug
 Something is broken

  
type
chore
 Maintenance / housekeeping

  
type
docs
 Documentation

  
type
epic
 High-level roadmap item

  
type
feature
 New capability

  
type
task
 Concrete unit of work

  
wontfix
Will not be actioned

Inga etiketter
area/backups
area/ci
area/control-panel
area/identity
area/infra
area/observability
area/payments
area/security
area/storage
area/web
blocked
needs-info
needs-triage
ready-for-implementation
type
bug
type
chore
type
docs
type
epic
type
feature
type
task
wontfix
Milstolpe
Rensa milstolpe
Inga objekt
Ingen milstolpe
Projekt
Rensa projekt
Inga objekt
Inget projekt
Bitborg Infrastructure
Tilldelade
Rensa tilldelade
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens
bitborg/bitborg-infra#40
Ingen beskrivning angiven.