Backups: add a second off-provider destination #41

Stängd
öppnade 2026-07-09 23:03:57 +00:00 av supernaut · 2 kommentarer
Ägare

Add a second backup destination outside the current provider for true provider-failure resilience.

Epic: gitborg/gitborg-docs#2

Add a second backup destination outside the current provider for true provider-failure resilience. Epic: gitborg/gitborg-docs#2
Upphovsperson
Ägare

Use Glesys as secondary S3 backup provider.

Information on pricing can be found on this page under the headline "Object Storage":
https://glesys.se/priser/

Documentation for usage:
https://docs.glesys.com/products/storage/object-storage

Use Glesys as secondary S3 backup provider. Information on pricing can be found on this page under the headline "Object Storage": https://glesys.se/priser/ Documentation for usage: https://docs.glesys.com/products/storage/object-storage
Upphovsperson
Ägare

Implemented, deployed, and verified live on prod. 🇸🇪

Glesys (Swedish, Falkenberg — Ceph S3) is now the primary off-site destination; Hetzner (hel1, EU) is the secondary cross-provider redundancy copy. Off-site targets are an ordered backup_s3_destinations list; each backup uploads to both with strict semantics (any destination failing fails the run); bucket names + keys are vaulted. Per-destination metric gitborg_backup_offsite_last_status{destination} + BackupOffsiteFailed (warning) alongside BackupFailed (critical). The restore drill (ADR 0027) now restores from the Swedish primary.

Verified end-to-end: a live backup landed the archive in both buckets (glesys=0, hetzner=0, overall=0); the restore drill fetched from Glesys and forgejo doctor confirmed 5 repos + DB consistency, VM torn down with no leak; the new alert is loaded (warning) and not firing.

This puts the primary off-site copy on Swedish soil, largely closing the principle-#1 exception — it partially satisfies #39's intent, but #39 stays open for Bahnhof S3 specifically (blocked on its DPA).

Implemented, deployed, and **verified live on prod**. 🇸🇪 **Glesys** (Swedish, Falkenberg — Ceph S3) is now the **primary** off-site destination; **Hetzner** (hel1, EU) is the **secondary** cross-provider redundancy copy. Off-site targets are an ordered `backup_s3_destinations` list; each backup uploads to both with **strict** semantics (any destination failing fails the run); bucket names + keys are vaulted. Per-destination metric `gitborg_backup_offsite_last_status{destination}` + `BackupOffsiteFailed` (warning) alongside `BackupFailed` (critical). The restore drill (ADR 0027) now restores from the Swedish primary. **Verified end-to-end:** a live backup landed the archive in **both** buckets (`glesys`=0, `hetzner`=0, overall=0); the restore drill fetched from **Glesys** and `forgejo doctor` confirmed 5 repos + DB consistency, VM torn down with no leak; the new alert is loaded (warning) and not firing. This puts the **primary** off-site copy on Swedish soil, largely closing the principle-#1 exception — it partially satisfies **#39**'s intent, but #39 stays open for **Bahnhof S3** specifically (blocked on its DPA). - gitborg/gitborg-infra#67 (Closes this) - gitborg/gitborg-docs#18 — ADR 0011 update + architecture + principles
Logga in för att delta i denna konversation.
Ingen milstolpe
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens
bitborg/bitborg-infra#41
Ingen beskrivning angiven.