bitborg/bitborg-infra

Bevaka

Avgrena 0

fix(monitoring): pin the reconciler at v1.4.0, and repair two metric names §7b missed #418

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/rename-7d-reconciler-tag in i main

2026-08-10 19:57:35 +00:00

supernaut kommenterad

2026-08-10 19:41:14 +00:00

Ägare

ADR 0039 §7d. Two things: the intended tag bump, and a defect in §7b (#414) found while finishing it.

The intended change

bitborg_reconciler_image_tag v1.3.0 → v1.4.0 — the release that renames the seven metrics the reconciler emits (bitborg-auth-reconciler#40, tagged and built). The five alert descriptions deliberately left on gitborg_reconciler_* in #414 now flip, since the image emitting the new names is pinned as of this commit.

The reconciler role already pulls on a tag change ("#63 pattern, mirroring the forgejo role"), so this is not the silent image-tag no-op — the new image lands on apply.

⚠️ §7b left two LIVE metric names behind

bitborg-monitoring-probe.sh.j2 emitted this, and #414 half-renamed it:

printf '# HELP bitborg_monitoring_vm_reachable …\n# TYPE bitborg_monitoring_vm_reachable gauge\ngitborg_monitoring_vm_reachable %s\n'

HELP and TYPE moved; the sample line did not. The textfile declared one name and emitted another. Same for gitborg_monitoring_alerting_healthy — which feeds the watchdog probe, so of the two that's the one that matters.

Cause: the rename used a (?<![A-Za-z0-9_]) lookbehind, and inside a single-quoted printf format string the \n escape puts a literal n immediately before the token, so the lookbehind refused to match. The mirror-image lookahead skipped four prose globs written gitborg_backup_drill_* and friends, where the trailing _ belongs to the glob.

Nothing broke in production — but only because the consumers are dual-brand and still match either name. A flat matcher would have blanked the watchdog silently.

Why my verification missed it

For the consumers I asserted "zero gitborg_ left inside any expr". For the producers I only counted replacements.

A count of what changed says nothing about what remains. The assertion has to be on the leftovers — this change adds it, and it now passes: no gitborg_ token survives in ansible/ except the three deliberately pinned non-metrics (gitborg_web the Postgres role, gitborg_ephemeral the OpenStack delete guard, gitborg_network_id a tofu output) and one sentence that describes the pre-apply state on purpose.

Verified

vmalert -dryRun PASS, 46 alerts, set identical to pre-§7a
the two repaired printf blocks show 3 occurrences each (HELP + TYPE + sample), so declaration and name agree
ansible-lint 0/0 across 194 files at production; pinned shellcheck clean

ADR 0039 §7d. Two things: the intended tag bump, and **a defect in §7b (#414)** found while finishing it. ## The intended change `bitborg_reconciler_image_tag` **v1.3.0 → v1.4.0** — the release that renames the seven metrics the reconciler emits (bitborg-auth-reconciler#40, tagged and built). The five alert descriptions deliberately left on `gitborg_reconciler_*` in #414 now flip, since the image emitting the new names is pinned as of this commit. The reconciler role already pulls on a tag change (*"#63 pattern, mirroring the forgejo role"*), so this is **not** the silent image-tag no-op — the new image lands on apply. ## ⚠️ §7b left two LIVE metric names behind `bitborg-monitoring-probe.sh.j2` emitted this, and #414 half-renamed it: ``` printf '# HELP bitborg_monitoring_vm_reachable …\n# TYPE bitborg_monitoring_vm_reachable gauge\ngitborg_monitoring_vm_reachable %s\n' ``` **HELP and TYPE moved; the sample line did not.** The textfile declared one name and emitted another. Same for `gitborg_monitoring_alerting_healthy` — which feeds the **watchdog** probe, so of the two that's the one that matters. Cause: the rename used a `(?<![A-Za-z0-9_])` lookbehind, and inside a single-quoted `printf` format string the `\n` escape puts a literal **`n`** immediately before the token, so the lookbehind refused to match. The mirror-image lookahead skipped four prose globs written `gitborg_backup_drill_*` and friends, where the trailing `_` belongs to the glob. Nothing broke in production — but only because the consumers are dual-brand and still match either name. A flat matcher would have blanked the watchdog silently. ## Why my verification missed it For the **consumers** I asserted *"zero `gitborg_` left inside any expr"*. For the **producers** I only counted replacements. A count of what changed says nothing about what remains. The assertion has to be on the leftovers — this change adds it, and it now passes: no `gitborg_` token survives in `ansible/` except the three deliberately pinned non-metrics (`gitborg_web` the Postgres role, `gitborg_ephemeral` the OpenStack delete guard, `gitborg_network_id` a tofu output) and one sentence that describes the pre-apply state on purpose. ## Verified - `vmalert -dryRun` **PASS**, 46 alerts, set identical to pre-§7a - the two repaired `printf` blocks show **3 occurrences each** (HELP + TYPE + sample), so declaration and name agree - `ansible-lint` 0/0 across 194 files at `production`; pinned `shellcheck` clean

supernaut lade till 1 incheckning

2026-08-10 19:41:14 +00:00

fix(monitoring): pin the reconciler at v1.4.0, and repair two metric names §7b missed

ci / ci (pull_request) Failing after 51s

Detaljer

3b061e142c

ADR 0039 §7d. Two things: the intended tag bump, and a defect in §7b
(#414) found while finishing it.

## The intended change

`bitborg_reconciler_image_tag` v1.3.0 -> v1.4.0, the release that renames
the seven metrics the reconciler emits. The five alert descriptions
deliberately left on `gitborg_reconciler_*` in #414 now flip, since the
image that emits the new names is pinned as of this commit.

The reconciler role already pulls on a tag change ("#63 pattern,
mirroring the forgejo role"), so this is not the silent image-tag no-op —
the new image lands on apply.

## §7b left two LIVE metric names behind

`bitborg-monitoring-probe.sh.j2` emitted this, and #414 half-renamed it:

  printf '# HELP bitborg_monitoring_vm_reachable ...
          # TYPE bitborg_monitoring_vm_reachable gauge
          \ngitborg_monitoring_vm_reachable %s\n'

HELP and TYPE moved; the SAMPLE line did not. The textfile would have
declared one name and emitted another. Same for
`gitborg_monitoring_alerting_healthy` — which feeds the watchdog probe,
so of the two this is the one that matters.

Cause: the rename used a `(?<![A-Za-z0-9_])` lookbehind, and inside a
single-quoted printf format string the `\n` escape puts a literal `n`
immediately before the token, so the lookbehind refused to match. The
mirror-image lookahead skipped four prose globs written
`gitborg_backup_drill_*` and friends, where the trailing `_` is part of
the glob.

Nothing broke in production, only because consumers are dual-brand and
still match either name. A flat matcher would have blanked the watchdog.

## Why the verification did not catch it

For the CONSUMERS I asserted "zero `gitborg_` left inside any expr". For
the PRODUCERS I only counted replacements. A count of what changed says
nothing about what remains — the assertion has to be on the leftovers.

This change adds that assertion and it now passes: no `gitborg_` token
survives in `ansible/` except the three deliberately pinned non-metrics
(`gitborg_web` the Postgres role, `gitborg_ephemeral` the OpenStack
delete guard, `gitborg_network_id` a tofu output) and one sentence that
describes the pre-apply state on purpose.

## Verified

`vmalert -dryRun` PASS with 46 alerts in a set identical to pre-§7a; the
two repaired printf blocks show 3 occurrences each (HELP + TYPE +
sample), so name and declaration agree; `ansible-lint` 0/0 across 194
files at `production`; pinned `shellcheck` clean.

supernaut kommenterad

2026-08-10 19:51:35 +00:00

Upphovsperson

Ägare

Pushed a second commit adding the CI check we discussed — and it earned its place before it was even wired up.

What it is

scripts/check-metric-names.py, gated on ansible or scripts changes. Four checks:

#	Check	Catches
1	no stray `gitborg_` outside a 3-entry allowlist	the leftover §7b should have asserted
2	no file carries one stem under both brands	a half-rename, with no allowlist to maintain
3	`# HELP x` ⟷ `# TYPE x` symmetry	malformed exposition
4	a declared name must actually be emitted	a rename onto a typo — invisible to 1 and 2

Two defects found immediately

scripts/load-web-image.sh — the break-glass manual deploy — could not have worked since §4b. It read a variable that no longer exists (registry host resolved empty), substituted that dead name when the image path had been decoupled onto bitborg_registry_host, and looked for the source in the pre-rename directory. Three faults in the one script nobody exercises until an emergency. Fixed here.

It caught me breaking the reconciler's Renovate annotation two commits earlier. My prose explaining the v1.4.0 bump went between the # renovate: line and its variable, which silently untracks the image — the precise regression check-renovate-annotations.py exists to catch, described in its own docstring as having happened to Forgejo. I committed it one screen below the warning. Prose moved above the annotation; all 7 annotations tracked again.

The checker had the same bug as the rename it guards

Its first version used a \b anchor and missed the real half-renamed printf — \ngitborg_x puts a literal n before the token, both are word characters, so no boundary exists. Identical to the lookbehind that caused the original incident.

The principle that fixes it is worth keeping: a rewriter must under-match, a detector must over-match. A false positive is visible and cheap to silence; a missed detection is silent forever. The \b is gone deliberately, with that reasoning inline.

Check 4 then false-positived on the real tree for a second self-inflicted reason — textfile writers build HELP, TYPE and the sample in one printf, so a set-membership test reads the sample as "the declaration again". It counts occurrences now.

Verified

Every check probed against input that must fail, plus two negative controls (clean multi-line, and clean all-on-one-printf-line). Seven probes, all as intended; real tree passes across 207 files.

I'd have shipped a check that passed on a broken tree if I hadn't probed it — which is the same lesson as the rest of this rename.

Pushed a second commit adding the CI check we discussed — and it earned its place before it was even wired up. ## What it is `scripts/check-metric-names.py`, gated on `ansible` **or** `scripts` changes. Four checks: | # | Check | Catches | | --- | --- | --- | | 1 | no stray `gitborg_` outside a 3-entry allowlist | the leftover §7b should have asserted | | 2 | no file carries one stem under **both** brands | a half-rename, with no allowlist to maintain | | 3 | `# HELP x` ⟷ `# TYPE x` symmetry | malformed exposition | | 4 | a declared name must actually be **emitted** | a rename onto a **typo** — invisible to 1 and 2 | ## Two defects found immediately **`scripts/load-web-image.sh` — the break-glass manual deploy — could not have worked since §4b.** It read a variable that no longer exists (registry host resolved *empty*), substituted that dead name when the image path had been decoupled onto `bitborg_registry_host`, and looked for the source in the pre-rename directory. Three faults in the one script nobody exercises until an emergency. Fixed here. **It caught me breaking the reconciler's Renovate annotation two commits earlier.** My prose explaining the v1.4.0 bump went **between** the `# renovate:` line and its variable, which silently untracks the image — the precise regression `check-renovate-annotations.py` exists to catch, described in its own docstring as having happened to Forgejo. I committed it one screen below the warning. Prose moved above the annotation; all 7 annotations tracked again. ## The checker had the same bug as the rename it guards Its first version used a `\b` anchor and **missed the real half-renamed printf** — `\ngitborg_x` puts a literal `n` before the token, both are word characters, so no boundary exists. Identical to the lookbehind that caused the original incident. The principle that fixes it is worth keeping: **a rewriter must under-match, a detector must over-match.** A false positive is visible and cheap to silence; a missed detection is silent forever. The `\b` is gone deliberately, with that reasoning inline. Check 4 then false-positived on the real tree for a second self-inflicted reason — textfile writers build HELP, TYPE and the sample in one `printf`, so a set-membership test reads the sample as "the declaration again". It counts occurrences now. ## Verified Every check probed against input that must fail, plus **two** negative controls (clean multi-line, and clean all-on-one-printf-line). Seven probes, all as intended; real tree passes across 207 files. I'd have shipped a check that passed on a broken tree if I hadn't probed it — which is the same lesson as the rest of this rename.

supernaut lade till 1 incheckning

2026-08-10 19:51:37 +00:00

feat(ci): assert the metric namespace is consistent, so a half-rename fails the build

ci / ci (pull_request) Successful in 1m40s

Detaljer

00cd34d817

The assertion §7b should have had. It verified the CONSUMERS by asserting
no `gitborg_` remained inside any expr, but verified the PRODUCERS by
counting replacements — and a count of what changed says nothing about
what remains. Two textfile writers were left half-renamed as a result.

`scripts/check-metric-names.py`, four checks:

  1. No stray `gitborg_` token outside a 3-entry allowlist of pinned
     non-metrics, each with its reason. Line-level opt-out requires a
     stated reason.
  2. No file carries the same metric stem under BOTH brands — what a
     half-rename looks like from outside, with no allowlist to maintain.
  3. Every `# HELP x` has a `# TYPE x` and vice versa.
  4. A declared name must actually be EMITTED. This is the one neither a
     leftover scan nor check 2 can see: rename a sample onto a TYPO and
     nothing is stale, both spellings are correctly branded, and the
     metric is silently wrong forever.

## Two defects it found immediately

`scripts/load-web-image.sh`, the BREAK-GLASS manual deploy path, could not
have worked since §4b: it read a variable that no longer exists (so the
registry host resolved empty), substituted that dead name when the image
path had been decoupled onto `bitborg_registry_host`, and looked for the
source in the pre-rename directory. Three faults in the one script nobody
exercises until an emergency.

And it caught me breaking the reconciler's Renovate annotation two
commits earlier: the prose explaining the v1.4.0 bump went BETWEEN the
`# renovate:` line and its variable, which silently untracks the image.
That is the precise regression `check-renovate-annotations.py` exists to
catch, and its docstring describes it happening to Forgejo — committed one
screen below the warning. Prose moved above the annotation; all 7 tracked
again.

## The checker had the same bug as the rename

Its first version used a `\b` anchor and missed the real half-renamed
printf, because `\ngitborg_x` puts a literal `n` before the token and both
are word characters, so no boundary exists. Identical to the lookbehind
that caused the incident.

The fix is a principle worth keeping: a REWRITER must under-match or it
corrupts unrelated text, but a DETECTOR must over-match — a false positive
is visible and cheap to silence, a missed detection is silent forever. The
`\b` is gone deliberately, with that reasoning recorded inline.

Check 4 then false-positived on the real tree for a second self-inflicted
reason: textfile writers build HELP, TYPE and the sample in ONE printf, so
a set-membership test reads the sample as "the declaration again". It
counts occurrences now.

Every check is verified against a probe that must fail plus two negative
controls (clean multi-line, and clean all-on-one-printf-line). All seven
probes behave as intended; the real tree passes across 207 files.

supernaut tvångsskickade feat/rename-7d-reconciler-tag från 00cd34d817

ci / ci (pull_request) Successful in 1m40s

Detaljer

till fade561aa6

ci / ci (pull_request) Successful in 1m47s

Detaljer

2026-08-10 19:54:17 +00:00

Jämför

supernaut sammanfogade incheckning 04e5657b67 till main

2026-08-10 19:57:35 +00:00

supernaut tog bort grenen feat/rename-7d-reconciler-tag

2026-08-10 19:57:35 +00:00