perf(backup): upload to off-site destinations in parallel #148

Sammanfogat

supernaut sammanfogade 1 incheckning från perf/backup-parallel-offsite in i main

2026-07-19 19:05:33 +00:00

supernaut kommenterad

2026-07-19 19:01:26 +00:00

Ägare

Why

The #147 verification run showed the off-site phase uploads destinations sequentially — two ~7.6 GB uploads (glesys + hetzner) at ~15 min each ≈ 30 min serial. They're independent full copies of the same ciphertext to different providers, so they can run concurrently.

Change

roles/backup/templates/bitborg-backup.sh.j2 — launch each destination's upload_to as a background job and collect exit statuses via wait. Off-site phase now ≈ the slowest destination (~15 min), not the sum.

upload_to now returns its status (a background job can't set a parent var like OFFSITE_FAILED); the parent maps PID→status and sets OFFSITE_FAILED from wait.
Archive mount :ro,Z → :ro,z: destinations mount ${ARCHIVE} read-only concurrently, so the SELinux relabel must be shareable (:Z is private/per-container and would conflict). No-op on this AppArmor host; correct if SELinux is ever enabled.
M7 semantics preserved: per-destination write_offsite_status is file-based (works across subshells), local success stays independent of off-site, BackupOffsiteFailed still fires per destination.

Validation

Functional test of launch/wait/collect: statuses correctly attributed per destination (both-ok→0, one-fails→1, both-fail→1); two 1 s jobs finish in 1 s (concurrent, not 2 s).
ansible-lint (production profile), --check --diff renders clean (0 failed), bash -n on the rendered script.
Live wall-clock to be confirmed on the next backup run after apply.

## Why The #147 verification run showed the off-site phase uploads destinations **sequentially** — two ~7.6 GB uploads (glesys + hetzner) at ~15 min each ≈ **30 min serial**. They're independent full copies of the same ciphertext to different providers, so they can run concurrently. ## Change `roles/backup/templates/bitborg-backup.sh.j2` — launch each destination's `upload_to` as a background job and collect exit statuses via `wait`. Off-site phase now ≈ the **slowest** destination (~15 min), not the sum. - `upload_to` now `return`s its status (a background job can't set a parent var like `OFFSITE_FAILED`); the parent maps PID→status and sets `OFFSITE_FAILED` from `wait`. - Archive mount `:ro,Z` → `:ro,z`: destinations mount `${ARCHIVE}` read-only **concurrently**, so the SELinux relabel must be shareable (`:Z` is private/per-container and would conflict). No-op on this AppArmor host; correct if SELinux is ever enabled. - M7 semantics preserved: per-destination `write_offsite_status` is file-based (works across subshells), local success stays independent of off-site, `BackupOffsiteFailed` still fires per destination. ## Validation - **Functional test** of launch/`wait`/collect: statuses correctly attributed per destination (both-ok→0, one-fails→1, both-fail→1); two 1 s jobs finish in 1 s (concurrent, not 2 s). - ansible-lint (production profile), `--check --diff` renders clean (0 failed), `bash -n` on the rendered script. - Live wall-clock to be confirmed on the next backup run after apply.

supernaut lade till 1 incheckning

2026-07-19 19:01:27 +00:00

perf(backup): upload to off-site destinations in parallel

ci / ci (pull_request) Successful in 3m17s

Detaljer

daecdf78b2

The off-site phase uploaded destinations sequentially — two ~7.6 GB
uploads (glesys + hetzner) at ~15 min each = ~30 min. They are
independent copies of the same ciphertext, so launch each upload_to as a
background job and collect statuses via wait (upload_to now returns its
status; the parent maps PID->status). Off-site phase now takes as long
as the slowest destination, not the sum.

Archive mount :ro,Z -> :ro,z so concurrent read-only mounts can share
the SELinux relabel (no-op on this AppArmor host). M7 semantics
preserved: per-destination metrics are file-based, local success stays
independent, BackupOffsiteFailed still fires per destination.

Validated: functional test of the wait/collect logic (per-destination
status + concurrency), ansible-lint (production), --check renders clean,
bash -n.

supernaut sammanfogade incheckning 7b91ae2a0d till main

2026-07-19 19:05:33 +00:00

supernaut tog bort grenen perf/backup-parallel-offsite

2026-07-19 19:05:33 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-19 19:05:34 +00:00

perf(backup): upload to off-site destinations in parallel (#148)

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-03 09:41:34 +00:00

perf(backup): upload to off-site destinations in parallel (#148)

Logga in för att delta i denna konversation.

Inga granskare