feat(health-check): post-apply health gate + validation/promotion docs (#107) #116

Sammanfogat

supernaut sammanfogade 1 incheckning från feat/107-post-apply-health-gate in i main

2026-07-18 21:42:25 +00:00

supernaut kommenterad

2026-07-18 21:38:40 +00:00

Ägare

Tier-2 of ADR 0030 — the last cheap piece of the pre-prod-validation epic (bitborg-docs#37). Turns the 2026-07-18 failure modes into an immediately-failing apply instead of a silent/lucky catch.

What it does

roles/health-check runs as the final, always-tagged role in each site.yml play, so every apply (full or --tags-scoped) ends by asserting, on the host it ran against:

core user services are active — the #94 dead-service class;
public endpoints serve via the local Caddy (--resolve, no NAT hairpin) — 2xx/3xx;
every *.prom in the textfile dir is world-readable — the #96 0600-mode class.

On any miss the apply fails loudly; no auto-rollback (fix-forward, per ADR 0030). All tasks are read-only (changed_when: false) and skipped under --check, so ansible-playbook site.yml --tags health-check --limit bitborg is a safe standalone probe.

Design note: node_exporter listens only on the podman network, so the metric check is done on disk (file mode) rather than via its HTTP endpoint — simpler, host-local, and it targets the exact #96 property.

Verified on prod, both directions

Happy path: services active, endpoints 2xx/3xx, .prom files readable → 0 failed.
Teeth: a bogus service name fails the gate; the file-mode check flags a 0600 .prom and passes at 0644.

Docs

Runbook: new Change validation & promotion (ADR 0030) section (trunk-based, branch protection, tiers, off-peak, rehearse-first) + rehearse-first note in the Forgejo upgrade policy.

Operator action (not code)

Enable branch protection on main in Forgejo (Settings → Branches → protect main, require the ci status check). That's the merge-time half of ADR 0030 and can't be set from the repo.

Closes #107.

Tier-2 of ADR 0030 — the last cheap piece of the pre-prod-validation epic (bitborg-docs#37). Turns the 2026-07-18 failure modes into an **immediately-failing apply** instead of a silent/lucky catch. ## What it does `roles/health-check` runs as the **final, `always`-tagged role** in each `site.yml` play, so every apply (full or `--tags`-scoped) ends by asserting, on the host it ran against: - core user services are `active` — the **#94** dead-service class; - public endpoints serve via the local Caddy (`--resolve`, no NAT hairpin) — 2xx/3xx; - every `*.prom` in the textfile dir is world-readable — the **#96** 0600-mode class. On any miss the apply **fails loudly**; **no auto-rollback** (fix-forward, per ADR 0030). All tasks are read-only (`changed_when: false`) and skipped under `--check`, so `ansible-playbook site.yml --tags health-check --limit bitborg` is a **safe standalone probe**. Design note: node_exporter listens only on the podman network, so the metric check is done **on disk (file mode)** rather than via its HTTP endpoint — simpler, host-local, and it targets the exact #96 property. ## Verified on prod, both directions - Happy path: services active, endpoints 2xx/3xx, `.prom` files readable → **0 failed**. - Teeth: a bogus service name **fails** the gate; the file-mode check flags a 0600 `.prom` and passes at 0644. ## Docs - Runbook: new **Change validation & promotion (ADR 0030)** section (trunk-based, branch protection, tiers, off-peak, rehearse-first) + rehearse-first note in the Forgejo upgrade policy. ## Operator action (not code) **Enable branch protection on `main`** in Forgejo (Settings → Branches → protect `main`, require the `ci` status check). That's the merge-time half of ADR 0030 and can't be set from the repo. Closes #107.

supernaut lade till 1 incheckning

2026-07-18 21:38:40 +00:00

feat(health-check): post-apply health gate + validation/promotion docs (#107 )

ci / ci (pull_request) Successful in 3m0s

Detaljer

dfec108c87

Tier-2 of ADR 0030 — turns the 2026-07-18 failure modes into an
immediately-failing apply instead of a silent/lucky catch.

- roles/health-check: final, always-tagged role in each site.yml play.
  Asserts core user services are active (the #94 dead-service class),
  public endpoints serve via the local Caddy (--resolve, no hairpin),
  and every *.prom in the textfile dir is world-readable (the #96
  0600-mode class). Halt + alert on any miss; NO auto-rollback
  (fix-forward, ADR 0030). Read-only (changed_when:false) + skipped
  under --check, so `--tags health-check` is a safe standalone probe.
  node_exporter is podman-network-only, so the metric check is done on
  disk (file mode) rather than via its HTTP endpoint.
- site.yml: wired into both the gitborg and monitoring plays with
  host-appropriate service/endpoint lists.
- runbook: new 'Change validation & promotion (ADR 0030)' section
  (trunk-based, branch protection, the tiers, off-peak, rehearse-first)
  + rehearse-first note in the Forgejo upgrade policy.

Verified on prod both directions: happy path green (services active,
endpoints 2xx/3xx, prom files readable); a bogus service name fails the
gate (teeth); the file-mode check flags a 0600 .prom and passes at 0644.

Branch protection on main is a Forgejo repo setting — operator action,
documented in the runbook.

Closes #107.