bitborg/bitborg-docs

Bevaka

Avgrena 0

Pre-prod validation: staging environment for infra changes #37

Nytt ärende

Stängd

öppnade 2026-07-18 15:00:12 +00:00 av supernaut · 1 kommentar

supernaut kommenterad

2026-07-18 15:00:12 +00:00

Ägare

Give infra changes a production-like place to prove themselves before they touch prod. The 2026-07-18 outage (caddy #94, ~4 min down) is the motivating case: ansible-playbook --check reported the change clean, but it can't actually start containers, so it missed a NoNewPrivileges + DropCapability=ALL interaction with a file-capability binary that crash-looped Caddy on the real apply.

Decided — ADR 0030 (accepted 2026-07-18)

Phased, cost-tiered. Trunk-based, single main (branch-protected on CI-green — configured; review soft), directory-scoped environments, not Gitflow.

Tier 0 — CI smoke (every PR): shipped (#104) — container-start + textfile-metric guards; catches the #94/#96 classes.
Tier 2 — post-apply health gate + branch protection + rehearse-first: shipped (#107) — final always-tagged play asserts core services active + endpoints serve + .prom files readable; halt+alert, fix-forward.
Tier 1 — ephemeral staging + env split: deferred/last-ditch (#103/#105/#106) — documented as the target shape, built only if a real-host-only failure demands it. May never be needed.
One hard rule: major Postgres/Forgejo (irreversible-forward) upgrades rehearsed first — restore-drill clone (ADR 0027) as first-choice vehicle before full staging.

Full rationale: bitborg-docs/decisions/0030-environments-and-promotion.md.

The now-scoped slice (Tier 0 + Tier 2) is complete. Tier 1 remains deferred by design.

Tasks

gitborg/gitborg-docs#38 — ADR: environments & promotion strategy — done (ADR 0030)
gitborg/gitborg-infra#104 — Tier 0: full-stack smoke (container-start + textfile-metric guards) — done (PRs #110, #112)
gitborg/gitborg-infra#107 — post-apply health gate + branch protection + runbook rehearse-first — done (PR #116)
gitborg/gitborg-infra#103 — env layering (inventory + group_vars) — deferred (with Tier-1)
gitborg/gitborg-infra#105 — Tier 1: ephemeral staging host — deferred / last-ditch
gitborg/gitborg-infra#106 — staging data seeding + web parity — deferred (with Tier-1)

Give infra changes a production-like place to prove themselves **before** they touch prod. The 2026-07-18 outage (caddy #94, ~4 min down) is the motivating case: `ansible-playbook --check` reported the change clean, but it can't actually start containers, so it missed a `NoNewPrivileges` + `DropCapability=ALL` interaction with a file-capability binary that crash-looped Caddy on the real apply. ## Decided — ADR 0030 (accepted 2026-07-18) Phased, cost-tiered. **Trunk-based, single `main`** (branch-protected on CI-green — configured; review soft), directory-scoped environments, **not Gitflow**. - **Tier 0 — CI smoke (every PR): shipped** (#104) — container-start + textfile-metric guards; catches the #94/#96 classes. - **Tier 2 — post-apply health gate + branch protection + rehearse-first: shipped** (#107) — final `always`-tagged play asserts core services active + endpoints serve + `.prom` files readable; halt+alert, fix-forward. - **Tier 1 — ephemeral staging + env split: deferred/last-ditch** (#103/#105/#106) — documented as the target shape, built only if a real-host-only failure demands it. May never be needed. - **One hard rule:** major Postgres/Forgejo (irreversible-forward) upgrades rehearsed first — restore-drill clone (ADR 0027) as first-choice vehicle before full staging. Full rationale: `bitborg-docs/decisions/0030-environments-and-promotion.md`. **The now-scoped slice (Tier 0 + Tier 2) is complete.** Tier 1 remains deferred by design. ### Tasks - [x] gitborg/gitborg-docs#38 — ADR: environments & promotion strategy — **done** (ADR 0030) - [x] gitborg/gitborg-infra#104 — Tier 0: full-stack smoke (container-start + textfile-metric guards) — **done** (PRs #110, #112) - [x] gitborg/gitborg-infra#107 — post-apply health gate + branch protection + runbook rehearse-first — **done** (PR #116) - [ ] gitborg/gitborg-infra#103 — env layering (inventory + group_vars) — **deferred** (with Tier-1) - [ ] gitborg/gitborg-infra#105 — Tier 1: ephemeral staging host — **deferred / last-ditch** - [ ] gitborg/gitborg-infra#106 — staging data seeding + web parity — **deferred** (with Tier-1)

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-07-18 15:01:09 +00:00

Environment layering: inventory + group_vars for prod/staging #103

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-07-18 15:01:09 +00:00

Tier 0: full-stack smoke test (Molecule) — assert every service container starts #104

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-07-18 15:01:09 +00:00

Tier 1: ephemeral OpenStack staging host (boot → site.yml → smoke → destroy) #105

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-07-18 15:01:10 +00:00

Staging data seeding + bitborg-web parity (no ADR-0019 drift) #106

supernaut refererade till detta ärende från bitborg/bitborg-infra

2026-07-18 15:01:10 +00:00

Manual-promotion workflow + post-apply prod health gate #107

supernaut refererade till detta ärende

2026-07-18 15:01:18 +00:00

ADR: environments & promotion strategy #38

supernaut lade till etiketterna

2026-07-18 15:02:19 +00:00

supernaut lade till detta till projektet Bitborg Roadmap

2026-07-18 15:13:23 +00:00

~~supernaut refererade till detta ärende 2026-07-18 21:24:12 +00:00~~

docs: ADR 0030 — environments & promotion strategy (bitborg-infra#38) #39

supernaut kommenterad

2026-09-24 13:05:10 +00:00

Upphovsperson

Ägare

Shipping the delivered tiers. Tier 0 and Tier 2 are done. Tier 1 is deferred until there is traffic that justifies a standing staging environment.

The Tier 1 tasks (bitborg/bitborg-infra#103, #105, #106) stay in the infra backlog. Reopen this or file a new epic when they are picked up.

Shipping the delivered tiers. Tier 0 and Tier 2 are done. Tier 1 is deferred until there is traffic that justifies a standing staging environment. The Tier 1 tasks (bitborg/bitborg-infra#103, #105, #106) stay in the infra backlog. Reopen this or file a new epic when they are picked up.

supernaut stängde ärendet

2026-09-24 13:05:11 +00:00

Logga in för att delta i denna konversation.

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-docs#37

Ingen beskrivning angiven.

Rader
Kolumner