#104 - Tier 0: full-stack smoke test (Molecule) — assert every service container starts - bitborg/bitborg-infra

supernaut kommenterad

2026-07-18 15:01:09 +00:00

Ägare

Tier 0 — the cheapest guard, and the one that would have caught #94. A CI job that converges the container roles and asserts each service actually starts and responds, not just that the templates render.

Molecule scenario (podman driver) converging postgres → forgejo → caddy → kanidm → web against a systemd-capable container/VM.
Assertions: each *.service is active; podman ps shows the container up (not crash-looping); HTTP smoke on caddy (all host blocks answer), forgejo /api/healthz, etc.
Explicitly reproduce the #94 class: a hardened unit (NoNewPrivileges=true + DropCapability=ALL) fronting an image whose binary carries file capabilities must still exec — a regression here must fail CI.
Runs on every bitborg-infra PR that touches ansible/.

Note the parity gap (rootless podman in CI ≠ Debian host for sysctls/pasta) — that's what Tier 1 covers; Tier 0 catches container-start/config failures for free.

Epic: gitborg/gitborg-docs#37

**Tier 0 — the cheapest guard, and the one that would have caught #94.** A CI job that converges the container roles and asserts each service *actually starts and responds*, not just that the templates render. - Molecule scenario (podman driver) converging postgres → forgejo → caddy → kanidm → web against a systemd-capable container/VM. - Assertions: each `*.service` is `active`; `podman ps` shows the container up (not crash-looping); HTTP smoke on caddy (all host blocks answer), forgejo `/api/healthz`, etc. - **Explicitly reproduce the #94 class**: a hardened unit (`NoNewPrivileges=true` + `DropCapability=ALL`) fronting an image whose binary carries file capabilities must still `exec` — a regression here must fail CI. - Runs on every bitborg-infra PR that touches `ansible/`. Note the parity gap (rootless podman in CI ≠ Debian host for sysctls/pasta) — that's what Tier 1 covers; Tier 0 catches container-start/config failures for free. Epic: gitborg/gitborg-docs#37

supernaut refererade till detta ärende från bitborg/bitborg-docs

2026-07-18 15:01:30 +00:00

Pre-prod validation: staging environment for infra changes #37

supernaut lade till etiketterna

2026-07-18 15:01:42 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-18 15:12:29 +00:00

supernaut refererade till detta ärende

2026-07-18 16:38:01 +00:00

feat(ci): Tier-0 smoke — hardened containers can start (#104) #110

supernaut refererade till detta ärende från en incheckning

2026-07-18 16:38:01 +00:00

feat(ci): Tier-0 smoke — hardened containers can start (#104)

supernaut kommenterad

2026-07-18 16:38:21 +00:00

Upphovsperson

Ägare

Increment 1 shipped — PR #110 (container-start smoke)

scripts/smoke-containers.py + a CI step: for each hardened Quadlet unit (NoNewPrivileges + DropCapability=ALL) with a public image, podman run it with the same caps and assert the entrypoint can exec — catching the #81/#94 class (file-cap binary crash-looping when a needed cap is dropped) that --check can't see. Runs on the host-backend runner (ADR 0021). Validated both directions locally: passes on the fixed units; fails when the #94 regression is re-introduced.

Feasibility note that shaped the scope: a full multi-service Molecule converge (as this issue originally sketched) needs systemd + rootless podman + podman-in-podman + the registry image + source-built Kanidm + the ADR-0025 fail-closed volume — flaky and costly for an every-PR tier, and it belongs on a real host. So Tier-0 = focused behavioural smoke of the actual failure classes; the full converge is Tier-1 (#105) on the ephemeral VM.

Remaining Tier-0 scope

Textfile-metric smoke (#96 class): run each .prom-writing script (or a factored core) and assert the output is world-readable (0644) and passes promtool check metrics. The token-audit.prom 0600 bug node_exporter silently rejected would be caught here.
(stretch) extend the container smoke to exercise a rendered Caddyfile / real config, not just entrypoint exec.

Full multi-service converge stays out of Tier-0 by design → #105.

### Increment 1 shipped — PR #110 (container-start smoke) `scripts/smoke-containers.py` + a CI step: for each hardened Quadlet unit (`NoNewPrivileges` + `DropCapability=ALL`) with a public image, `podman run` it with the same caps and assert the entrypoint can exec — catching the **#81/#94 class** (file-cap binary crash-looping when a needed cap is dropped) that `--check` can't see. Runs on the host-backend runner (ADR 0021). Validated both directions locally: passes on the fixed units; **fails** when the #94 regression is re-introduced. Feasibility note that shaped the scope: a full multi-service Molecule converge (as this issue originally sketched) needs systemd + rootless podman + podman-in-podman + the registry image + source-built Kanidm + the ADR-0025 fail-closed volume — flaky and costly for an every-PR tier, and it belongs on a real host. So Tier-0 = focused behavioural smoke of the actual failure classes; the full converge is **Tier-1 (#105)** on the ephemeral VM. ### Remaining Tier-0 scope - [ ] **Textfile-metric smoke (#96 class):** run each `.prom`-writing script (or a factored core) and assert the output is world-readable (0644) and passes `promtool check metrics`. The `token-audit.prom` 0600 bug node_exporter silently rejected would be caught here. - [ ] (stretch) extend the container smoke to exercise a rendered Caddyfile / real config, not just entrypoint exec. Full multi-service converge stays out of Tier-0 by design → #105.

supernaut refererade till detta ärende från en incheckning

2026-07-18 16:41:28 +00:00

feat(ci): Tier-0 smoke — hardened containers can start (#104) (#110)

supernaut refererade till detta ärende

2026-07-18 16:53:17 +00:00

feat(ci): Tier-0 textfile-metric safety guard (#104) #111

supernaut refererade till detta ärende från en incheckning

2026-07-18 16:53:17 +00:00

feat(ci): Tier-0 textfile-metric safety guard (#104)

supernaut kommenterad

2026-07-18 16:56:58 +00:00

Upphovsperson

Ägare

Increment 2 — PR #111 (textfile-metric guard)

The remaining #96-class item is done: scripts/smoke-textfile.py fails any metric script that publishes its .prom from a mktemp (0600) file without a world-readable chmod — traced to the actual mv source, so it's precise (no false positives on the 9 current scripts; teeth confirmed by reverting token-audit's chmod). Chosen static over behavioural on purpose — the faithful node_textfile_scrape_error==0 check needs a real converge and belongs in Tier-1 / the post-apply gate (#105 / #107).

Tier-0 status: complete

Both prod incident classes this week are now guarded in CI:

container-start under hardening (#94) — PR #110
textfile-metric readability (#96) — PR #111

Full multi-service integration converge remains out of Tier-0 by design → Tier-1 (#105). So this issue can close once #111 merges; #105 carries the converge.

Related: PR #112 gates all CI steps by changed area (docs-only PRs skip the infra toolchain) with the directional ansible↔opentofu cross-dependency handled — separate hygiene improvement, stacked on #111.

### Increment 2 — PR #111 (textfile-metric guard) The remaining #96-class item is done: `scripts/smoke-textfile.py` fails any metric script that publishes its `.prom` from a `mktemp` (0600) file without a world-readable `chmod` — traced to the actual `mv` source, so it's precise (no false positives on the 9 current scripts; teeth confirmed by reverting token-audit's chmod). Chosen static over behavioural on purpose — the faithful `node_textfile_scrape_error==0` check needs a real converge and belongs in Tier-1 / the post-apply gate (#105 / #107). ### Tier-0 status: complete Both prod incident classes this week are now guarded in CI: - [x] container-start under hardening (#94) — PR #110 - [x] textfile-metric readability (#96) — PR #111 Full multi-service integration converge remains **out of Tier-0 by design** → Tier-1 (#105). So this issue can close once #111 merges; #105 carries the converge. Related: PR #112 gates all CI steps by changed area (docs-only PRs skip the infra toolchain) with the directional ansible↔opentofu cross-dependency handled — separate hygiene improvement, stacked on #111.

supernaut kommenterad

2026-07-18 19:34:13 +00:00

Upphovsperson

Ägare

Tier-0 is complete and in main:

container-start smoke (#94 class) — #110
textfile-metric guard (#96 class) — merged via #112 (it carried #111's commit; #111 closed as superseded)

Both prod incident classes from this week are now guarded on every applicable PR, and CI only runs the infra toolchain when the relevant area changed (#112). The full multi-service integration converge stays out of Tier-0 by design → Tier-1 (#105). Closing; #105 carries the remainder.

Tier-0 is complete and in `main`: - container-start smoke (#94 class) — #110 - textfile-metric guard (#96 class) — merged via #112 (it carried #111's commit; #111 closed as superseded) Both prod incident classes from this week are now guarded on every applicable PR, and CI only runs the infra toolchain when the relevant area changed (#112). The full multi-service integration converge stays out of Tier-0 by design → **Tier-1 (#105)**. Closing; #105 carries the remainder.

supernaut stängde ärendet

2026-07-18 19:34:14 +00:00

supernaut refererade till detta ärende

2026-07-19 09:23:26 +00:00

smoke-containers: pull base images from the in-instance registry mirror (CI resilience + sovereignty) #137

supernaut refererade till detta ärende

2026-07-29 21:59:05 +00:00

fix(caddy): ride out the web container restart instead of 502ing #249

supernaut refererade till detta ärende från en incheckning

2026-07-29 22:02:30 +00:00

fix(caddy): ride out the web container restart instead of 502ing (#249)

supernaut refererade till detta ärende från en incheckning