backup: decouple Forgejo [storage] tier into incremental restic backups #211

Sammanfogat

supernaut sammanfogade 10 incheckningar från feat/backup-storage-decoupling in i main

2026-07-23 14:22:52 +00:00

supernaut kommenterad

2026-07-23 14:07:37 +00:00

Ägare

What

The daily encrypted backup archive bundled the entire Forgejo [storage] tier (~9 GB, dominated by
the package/container registry) into every run, so each archive grew to ~9.6 GB and the backup
volume filled up. This branch decouples that tier:

Daily age archive slimmed to DB + repos + kanidm (~0.5 GB) — the [storage] tier is no longer
in it.
New bitborg-backup-storage service + timer backs up [storage] separately with restic —
incremental, deduplicated, encrypted — to the off-site Glesys S3 repo. It never consumes the local
backup volume.
Excludes the regenerable/ephemeral subsystems (repo-archive, actions_log,
actions_artifacts, lost+found) — durable user data only.
New alerts BackupStorageFailed / BackupStorageStale, mirroring the existing backup alerts.
Restore reassembly: the restore drill and runbook now restore the age archive (DB/repos/
kanidm) and then restic restore the [storage] tier, with a fallback for legacy archives during
the transition window.

Before applying (do NOT apply to prod until these are done)

Set vault_backup_restic_password in the vault — backup.env will not render until it is set
(same bare-vault-ref convention as the other backup secrets).
Apply scoped: --tags backup,monitoring. First storage run does restic init + one full
snapshot.
Confirm the first storage snapshot, then run a restore rehearsal (drill) proving the
reassembled restore (age archive + restic restore) before relying on it.

Notes

restic image is pinned (Renovate-tracked); secrets stay in the 0600 EnvironmentFile and are passed
to podman as bare -e VAR (never on a command line).
The storage restic repo is off-site only (Glesys); retention --keep-daily 7 --keep-weekly 4.
⚠️ The drill's restore-on-scratch.sh [storage] section also changes on the rootless-migration
branch — reconcile the two on merge.

## What The daily encrypted backup archive bundled the entire Forgejo `[storage]` tier (~9 GB, dominated by the package/container registry) into every run, so each archive grew to ~9.6 GB and the backup volume filled up. This branch decouples that tier: - **Daily age archive slimmed** to DB + repos + kanidm (~0.5 GB) — the `[storage]` tier is no longer in it. - **New `bitborg-backup-storage` service + timer** backs up `[storage]` separately with **restic** — incremental, deduplicated, encrypted — to the off-site Glesys S3 repo. It never consumes the local backup volume. - Excludes the regenerable/ephemeral subsystems (`repo-archive`, `actions_log`, `actions_artifacts`, `lost+found`) — durable user data only. - **New alerts** `BackupStorageFailed` / `BackupStorageStale`, mirroring the existing backup alerts. - **Restore reassembly:** the restore drill and runbook now restore the age archive (DB/repos/ kanidm) and then `restic restore` the `[storage]` tier, with a fallback for legacy archives during the transition window. ## Before applying (do NOT apply to prod until these are done) - [x] Set `vault_backup_restic_password` in the vault — `backup.env` will not render until it is set (same bare-vault-ref convention as the other backup secrets). - [ ] Apply scoped: `--tags backup,monitoring`. First storage run does `restic init` + one full snapshot. - [ ] Confirm the first storage snapshot, then run a restore rehearsal (drill) proving the reassembled restore (age archive + `restic restore`) before relying on it. ## Notes - restic image is pinned (Renovate-tracked); secrets stay in the 0600 EnvironmentFile and are passed to `podman` as bare `-e VAR` (never on a command line). - The storage restic repo is off-site only (Glesys); retention `--keep-daily 7 --keep-weekly 4`. - ⚠️ The drill's `restore-on-scratch.sh` `[storage]` section also changes on the rootless-migration branch — reconcile the two on merge.

supernaut tvångsskickade feat/backup-storage-decoupling från 11bb90271b

ci / ci (pull_request) Successful in 1m26s

Detaljer

till cf9ecfc89a

ci / ci (pull_request) Successful in 1m26s

Detaljer

2026-07-23 14:09:54 +00:00

Jämför

supernaut lade till 1 incheckning

2026-07-23 14:20:22 +00:00

feat(ansible): add restic password & update glesys s3 credentials

ci / ci (pull_request) Successful in 1m21s