feat(backup): Glesys primary + Hetzner secondary off-site #67

Sammanfogat

supernaut sammanfogade 9 incheckningar från backup-glesys-primary in i main

2026-07-12 06:34:43 +00:00

supernaut kommenterad

2026-07-11 22:37:58 +00:00

Ägare

What

Adds Glesys (Swedish, Falkenberg — Ceph/RGW S3) as the primary off-site backup destination and demotes Hetzner (hel1, EU) to a secondary cross-provider redundancy copy. This satisfies #41 (a second off-provider destination) and advances the sovereignty goal: the primary off-site copy — the one the restore drill pulls from and a real recovery starts from — now lives on Swedish soil, largely closing the principle-#1 exception (ADR 0011).

Off-site targets are now an ordered backup_s3_destinations list (index 0 = primary). Non-secret S3 shape in group_vars; bucket names + keys are vaulted (repo is public).
Each backup uploads to all destinations with strict semantics — any destination failing fails the run. Per-destination metric gitborg_backup_offsite_last_status{destination} + BackupOffsiteFailed (warning) alongside the existing BackupFailed (critical).
The restore drill (ADR 0027) fetches from the primary (Glesys).
Built subagent-driven: 6 tasks, per-task review + a final whole-branch review.

See ADR 0011 update + architecture (both mermaid diagrams) + principles in gitborg/gitborg-docs#18 (companion PR), and the runbook section here.

Verified live on prod

Manual backup: archive landed in both buckets — gitborg_backup_offsite_last_status{glesys}=0, {hetzner}=0, overall gitborg_backup_last_run_status=0.
Restore drill: fetched the latest archive from Glesys, forgejo doctor confirmed 5 repos + DB consistency, VM torn down (no leak), gitborg_backup_drill_last_run_status=0.
BackupOffsiteFailed loaded (warning) and not firing; both per-destination series scraped into VictoriaMetrics.

Two review-caught fixes: a bash local a=$1 b=${a} evaluation bug (would crash under set -u); alert severity aligned to the design (warning, not critical) to avoid double-critical paging.

Epic: gitborg/gitborg-docs#2
Closes #41

## What Adds **Glesys** (Swedish, Falkenberg — Ceph/RGW S3) as the **primary** off-site backup destination and demotes **Hetzner** (hel1, EU) to a secondary cross-provider redundancy copy. This satisfies #41 (a second off-provider destination) **and** advances the sovereignty goal: the primary off-site copy — the one the restore drill pulls from and a real recovery starts from — now lives on Swedish soil, largely closing the principle-#1 exception (ADR 0011). - Off-site targets are now an ordered `backup_s3_destinations` list (index 0 = primary). Non-secret S3 shape in group_vars; **bucket names + keys are vaulted** (repo is public). - Each backup uploads to **all** destinations with **strict** semantics — any destination failing fails the run. Per-destination metric `gitborg_backup_offsite_last_status{destination}` + `BackupOffsiteFailed` (warning) alongside the existing `BackupFailed` (critical). - The restore drill (ADR 0027) fetches from the **primary** (Glesys). - Built subagent-driven: 6 tasks, per-task review + a final whole-branch review. See ADR 0011 update + architecture (both mermaid diagrams) + principles in gitborg/gitborg-docs#18 (companion PR), and the runbook section here. ## Verified live on prod - Manual backup: archive landed in **both** buckets — `gitborg_backup_offsite_last_status{glesys}=0`, `{hetzner}=0`, overall `gitborg_backup_last_run_status=0`. - Restore drill: fetched the latest archive **from Glesys**, `forgejo doctor` confirmed **5 repos + DB consistency**, VM torn down (no leak), `gitborg_backup_drill_last_run_status=0`. - `BackupOffsiteFailed` loaded (warning) and not firing; both per-destination series scraped into VictoriaMetrics. Two review-caught fixes: a bash `local a=$1 b=${a}` evaluation bug (would crash under `set -u`); alert severity aligned to the design (warning, not critical) to avoid double-critical paging. Epic: gitborg/gitborg-docs#2 Closes #41

supernaut lade till 8 incheckningar

2026-07-11 22:37:58 +00:00

feat(backup): model off-site destinations as an ordered list (#41 ) 772f863e7a