bitborg/bitborg-infra

Bevaka

Avgrena 0

feat(ci): Tier-0 textfile-metric safety guard (#104) #111

Stängd

supernaut vill sammanfoga 0 incheckningar från s[2]s in i main

supernaut kommenterad

2026-07-18 16:53:17 +00:00

Ägare

Second increment of the pre-prod validation epic (bitborg-docs#37) — closes the #96 failure class in CI.

The bug it catches

token-audit.prom was created by mktemp (mode 0600) then mv'd into the textfile dir, so node_exporter (non-root inside the rootless container) couldn't open it — node_textfile_scrape_error=1, metrics silently never reached VictoriaMetrics, despite a green apply and a status-0 run.

What it does

scripts/smoke-textfile.py (pure stdlib, runs anywhere): for each script that publishes a .prom into the node_exporter textfile dir, fail if the temp it mv's to the .prom comes from mktemp with no world-readable chmod. It traces the actual mv source, so scripts that publish via > (0644 via umask) and use mktemp only for unrelated work files are not flagged.

Why static, not behavioural

The 9 metric scripts call live curl/jq/GNU-date/podman/skopeo/openstack and rely on GNU date -d — a faithful stubbed run is more fragile than the deterministic file-mode bug it would catch (and can't be self-validated on the macOS control node). The faithful behavioural check — assert node_textfile_scrape_error==0 after a real converge — is Tier-1 / the post-apply health gate (#105 / #107), which is exactly what I ran by hand during the #96 incident.

Validated both directions

Positive: all 9 current metric scripts pass — no false positives. (An earlier coarse "contains mktemp" heuristic false-flagged 3 scripts that actually publish via >; I verified each and tightened the check to trace the real mv source.)
Negative (teeth): reverting token-audit's chmod 0644 makes the guard FAIL with a non-zero exit.

Together with #110 (container-start smoke), Tier-0 now covers both prod incident classes this week. Remaining #104 note: the full multi-service converge stays Tier-1 (#105).

Refs #104.

Second increment of the pre-prod validation epic (bitborg-docs#37) — closes the **#96** failure class in CI. ## The bug it catches `token-audit.prom` was created by `mktemp` (mode 0600) then `mv`'d into the textfile dir, so node_exporter (non-root inside the rootless container) couldn't open it — `node_textfile_scrape_error=1`, metrics silently never reached VictoriaMetrics, despite a green apply and a status-0 run. ## What it does `scripts/smoke-textfile.py` (pure stdlib, runs anywhere): for each script that publishes a `.prom` into the node_exporter textfile dir, fail if the temp it `mv`'s to the `.prom` comes from `mktemp` with no world-readable `chmod`. It **traces the actual mv source**, so scripts that publish via `>` (0644 via umask) and use `mktemp` only for unrelated work files are not flagged. ## Why static, not behavioural The 9 metric scripts call live `curl`/`jq`/GNU-`date`/`podman`/`skopeo`/`openstack` and rely on GNU `date -d` — a faithful stubbed run is more fragile than the deterministic file-mode bug it would catch (and can't be self-validated on the macOS control node). The faithful behavioural check — assert `node_textfile_scrape_error==0` after a real converge — is Tier-1 / the post-apply health gate (#105 / #107), which is exactly what I ran by hand during the #96 incident. ## Validated both directions - **Positive:** all 9 current metric scripts pass — no false positives. (An earlier coarse "contains mktemp" heuristic false-flagged 3 scripts that actually publish via `>`; I verified each and tightened the check to trace the real `mv` source.) - **Negative (teeth):** reverting token-audit's `chmod 0644` makes the guard FAIL with a non-zero exit. Together with #110 (container-start smoke), Tier-0 now covers **both** prod incident classes this week. Remaining #104 note: the full multi-service converge stays Tier-1 (#105). Refs #104.

supernaut lade till 1 incheckning

2026-07-18 16:53:17 +00:00

feat(ci): Tier-0 textfile-metric safety guard (#104 )

ci / ci (pull_request) Successful in 2m53s

Detaljer

0b37bc6f7f

Second increment of the pre-prod validation epic (gitborg-docs#37):
close the #96 failure class in CI. That bug — token-audit.prom created
by mktemp (0600) then mv'd into the textfile dir — was unreadable by
node_exporter (non-root in the rootless container), so metrics silently
never reached VM despite a green apply and a status-0 run.

scripts/smoke-textfile.py (pure stdlib, runs anywhere): for each script
that publishes a .prom into the node_exporter textfile dir, fail if the
temp it mv's to the .prom comes from mktemp with no world-readable chmod.
It traces the actual mv source, so scripts that publish via '>' (0644 via
umask) and use mktemp only for unrelated work files are not flagged.

Chosen static over behavioural deliberately: the 9 metric scripts call
live curl/jq/GNU-date/podman/skopeo/openstack and use GNU date -d, so a
faithful stubbed run is more fragile than the deterministic file-mode bug
it would catch. The faithful behavioural check — assert
node_textfile_scrape_error==0 after a real converge — is Tier-1 / the
post-apply health gate (#105 / #107).

Validated both directions: all 9 current scripts pass (no false positives
after tracing the real mv source — an earlier coarse 'contains mktemp'
heuristic false-flagged 3 that write via '>'); reverting token-audit's
chmod makes it FAIL. Wired into ci.yml.

Refs #104.

supernaut refererade till denna ändringsförfrågan

2026-07-18 16:56:31 +00:00

ci: run infra steps only when their area changed (+ ansible↔tofu x-dep) #112

supernaut refererade till denna ändringsförfrågan

2026-07-18 16:56:59 +00:00

Tier 0: full-stack smoke test (Molecule) — assert every service container starts #104

supernaut tvångsskickade feat/104-textfile-smoke från 0b37bc6f7f

ci / ci (pull_request) Successful in 2m53s

Detaljer

till d0097d2d39

ci / ci (pull_request) Successful in 2m57s

Detaljer

2026-07-18 18:54:47 +00:00

Jämför

supernaut kommenterad

2026-07-18 19:33:19 +00:00

Upphovsperson

Ägare

Superseded by #112 — no action needed. #112 was stacked on this branch, so its squash-merge carried this commit's content into main along with it. Verified: git diff main feat/104-textfile-smoke is empty — the textfile smoke (scripts/smoke-textfile.py), the smoke:textfile pnpm script, and the CI step are all present in main (via #112 / d0097d2). Closing without merging so it doesn't re-apply an empty change.

Superseded by #112 — no action needed. #112 was stacked on this branch, so its squash-merge carried this commit's content into `main` along with it. Verified: `git diff main feat/104-textfile-smoke` is empty — the textfile smoke (`scripts/smoke-textfile.py`), the `smoke:textfile` pnpm script, and the CI step are all present in `main` (via #112 / d0097d2). Closing without merging so it doesn't re-apply an empty change.

supernaut stängde denna ändringsförfrågan

2026-07-18 19:33:19 +00:00

supernaut refererade till denna ändringsförfrågan

2026-07-18 19:34:13 +00:00

Tier 0: full-stack smoke test (Molecule) — assert every service container starts #104

ci / ci (pull_request) Successful in 2m57s

Obligatorisk

Detaljer

Ändringsförfrågan stängd

Denna ändringsförfrågan kan inte öppnas eftersom grenen har tagits bort.

Logga in för att delta i denna konversation.

Inga granskare

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra!111

Ingen beskrivning angiven.

Rader
Kolumner