bitborg/bitborg-infra

Bevaka

Avgrena 0

Manual-promotion workflow + post-apply prod health gate #107

Nytt ärende

Stängd

öppnade 2026-07-18 15:01:10 +00:00 av supernaut · 1 kommentar

supernaut kommenterad

2026-07-18 15:01:10 +00:00

Ägare

Tier 2 — process + a safety net on prod itself.

Document the promotion flow in the runbook: trunk-based, merge to main → apply + smoke on ephemeral staging (for risky changes) → human-approved prod apply. No env branches.
Add a post-apply health gate to the prod apply flow: after site.yml, assert containers are active + public endpoints serve (extend the existing running-image verify into a real smoke check), and halt loudly / auto-recover if not — so a bad apply is caught in seconds, not by users. (The #94 verify failed after the outage; this makes the check block the failure mode earlier and pairs with the migration choreography.)
Bake "stage risky infra changes before prod" into the infra-apply skill + runbook, with a short checklist of what counts as risky (socket/port/capability/network/app.ini/image-tag).

Depends on Tier 0 (smoke assertions) and Tier 1 (staging target).

Epic: gitborg/gitborg-docs#37

**Tier 2 — process + a safety net on prod itself.** - Document the promotion flow in the runbook: trunk-based, merge to `main` → apply + smoke on ephemeral staging (for risky changes) → **human-approved** prod apply. No env branches. - Add a **post-apply health gate** to the prod apply flow: after `site.yml`, assert containers are `active` + public endpoints serve (extend the existing running-image verify into a real smoke check), and **halt loudly / auto-recover** if not — so a bad apply is caught in seconds, not by users. (The #94 verify failed *after* the outage; this makes the check block the failure mode earlier and pairs with the migration choreography.) - Bake "stage risky infra changes before prod" into the `infra-apply` skill + runbook, with a short checklist of what counts as risky (socket/port/capability/network/app.ini/image-tag). Depends on Tier 0 (smoke assertions) and Tier 1 (staging target). Epic: gitborg/gitborg-docs#37

supernaut refererade till detta ärende från bitborg/bitborg-docs

2026-07-18 15:01:30 +00:00

Pre-prod validation: staging environment for infra changes #37

supernaut lade till etiketterna

2026-07-18 15:01:42 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-18 15:12:29 +00:00

supernaut refererade till detta ärende

2026-07-18 16:53:17 +00:00

feat(ci): Tier-0 textfile-metric safety guard (#104) #111

supernaut refererade till detta ärende från en incheckning

2026-07-18 16:53:17 +00:00

feat(ci): Tier-0 textfile-metric safety guard (#104)

supernaut refererade till detta ärende

2026-07-18 16:56:59 +00:00

Tier 0: full-stack smoke test (Molecule) — assert every service container starts #104

supernaut lade till

ready-for-implementation

och tog bort

needs-triage

etiketter

2026-07-18 21:25:56 +00:00

supernaut kommenterad

2026-07-18 21:25:57 +00:00

Upphovsperson

Ägare

Scope confirmed by ADR 0030 (accepted) — this is the near-term build (Tier-2). Three parts:

Post-apply health gate — a final play in site.yml, host-scoped core-liveness: bitborg host = forgejo/postgres/caddy/kanidm/web containers active + www/git/auth serve + node_textfile_scrape_error==0; monitoring host = its containers + health endpoints. Halt + alert on any miss; no auto-rollback (fix-forward). Excludes the standing monitoring alert set (backup volume/drill) to avoid false-fails. Skipped under --check.
main branch protection — require the ci workflow green before merge (review soft — solo operator).
Runbook — the rehearse-first rule for major Postgres/Forgejo upgrades (restore-drill clone first), and the trunk-based/gated-promotion note.

(The Tier-1 staging + env-layering pieces this issue used to imply are deferred — see #103/#105/#106.)

Scope confirmed by ADR 0030 (accepted) — this is the near-term build (Tier-2). Three parts: 1. **Post-apply health gate** — a final play in `site.yml`, **host-scoped core-liveness**: bitborg host = `forgejo`/`postgres`/`caddy`/`kanidm`/`web` containers `active` + www/git/auth serve + `node_textfile_scrape_error==0`; monitoring host = its containers + health endpoints. **Halt + alert on any miss; no auto-rollback** (fix-forward). Excludes the standing monitoring alert set (backup volume/drill) to avoid false-fails. Skipped under `--check`. 2. **`main` branch protection** — require the `ci` workflow green before merge (review soft — solo operator). 3. **Runbook** — the rehearse-first rule for major Postgres/Forgejo upgrades (restore-drill clone first), and the trunk-based/gated-promotion note. (The Tier-1 staging + env-layering pieces this issue used to imply are deferred — see #103/#105/#106.)

supernaut refererade till detta ärende från en incheckning

2026-07-18 21:38:39 +00:00

feat(health-check): post-apply health gate + validation/promotion docs (#107)

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-07-18 21:38:40 +00:00

feat(health-check): post-apply health gate + validation/promotion docs (#107) #116

supernaut stängde ärendet

2026-07-18 21:42:25 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-18 21:42:26 +00:00

feat(health-check): post-apply health gate + validation/promotion docs (#107) (#116)

supernaut refererade till detta ärende