ci: validate the vmalert rule set before it can reach production #440

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/validate-alert-rules in i main

2026-08-18 19:15:37 +00:00

supernaut kommenterad

2026-08-18 18:53:40 +00:00

Ägare

Closes #439.

The gap

The 49 rules in roles/monitoring/templates/alert-rules.yml.j2 had no gate. The Render alert rules task carries no validate: and notifies Restart vmalert directly, so a typo went straight
to production. grep -rn promtool over the repo returned nothing.

Why it is worse than one broken alert

vmalert parses the rule file as a whole, and vmalert.container.j2 does not pass
-rule.validateExpressions=false. One unbalanced paren is fatal to all 49 rules, not to one, and
Restart=always makes that a crash loop rather than a stopped unit. Measured against the pinned
v1.147.0:

# real startup, no -dryRun
fatal  app/vmalert/main.go:167  cannot parse configuration file: failed to parse [rules.yml]
exit 255

Nothing would notice. The Watchdog dead-man's switch exists, but its Alertmanager receiver
deliberately has no integrations yet, so it only protects against vmalert death once an external
monitor consumes it and alarms on silence.

Shape

scripts/check-alert-rules.py, wired into CI on the ansible or scripts change gates.

Render with the real variables under StrictUndefined. All 34 variables the template uses
resolve with no vault password: 30 from the monitoring role's defaults, 4 from plaintext
group_vars/all/vars.yml, plus ansible_managed. So nothing needs faking, and strictness is the
point. Substituting a placeholder for an undefined variable renders > 1, which parses, so the
check would pass on a role default renamed out from under the template. That is the most likely
failure of all.

Parse with vmalert, not promtool. promtool works on today's file but is the wrong engine:
vmalert is VictoriaMetrics and accepts MetricsQL that promtool rejects, so promtool could only
approximate and would drift toward false failures. -dryRun hits the same config parse the real
startup does and needs no datasource. The tag is read from the role defaults, so no second literal
can drift.

PyYAML and Jinja2 both arrive with the ansible baked into the runner image (#65), so nothing new
is installed.

The self-test earned its keep immediately

--self-test mutates the inputs and asserts the check fails. It runs in CI, not just once by hand.

On its first run it failed, and it was right to. A threshold set to null renders the literal string
None, and x > None is legal PromQL, because None parses as a vector selector. vmalert accepted
the file and the rule could never have fired. StrictUndefined did not help either, because the key
existed. Fixed by dropping null values at the merge step so the strict render rejects them.

self-test, proving the check is not vacuous:
  ok   unmodified tree passes
  ok   unbalanced paren in a rule expression is caught
  ok   a role default renamed out from under the template is caught
  ok   a variable emptied to a blank value is caught

self-test: 4/4. The check fails when it should

Verification

./scripts/check-alert-rules.py reports OK: 49 vmalert rules render and parse, matching the 49
- alert: lines in the template.
./scripts/check-alert-rules.py --self-test passes 4/4, and each case was observed failing before
the corresponding guard existed.
ruff, prettier, markdownlint, ansible-lint, shellcheck and gitleaks all clean.

Not in scope

The vmalert image is not in registry_mirror_images, so this step pulls from docker.io. That
leaves it as one of the few CI steps still reaching outside the instance (#137). Mirroring it is a
cheap follow-up.
Giving the Watchdog heartbeat an external consumer is the larger gap, and is what makes a
vmalert failure silent today. Not tracked yet.
This is the static half only. The functional Tier-0 smoke story stays #104 / bitborg-docs#37.

Closes #439. ## The gap The 49 rules in `roles/monitoring/templates/alert-rules.yml.j2` had no gate. The `Render alert rules` task carries no `validate:` and notifies `Restart vmalert` directly, so a typo went straight to production. `grep -rn promtool` over the repo returned nothing. ## Why it is worse than one broken alert vmalert parses the rule file as a whole, and `vmalert.container.j2` does not pass `-rule.validateExpressions=false`. One unbalanced paren is fatal to all 49 rules, not to one, and `Restart=always` makes that a crash loop rather than a stopped unit. Measured against the pinned v1.147.0: ``` # real startup, no -dryRun fatal app/vmalert/main.go:167 cannot parse configuration file: failed to parse [rules.yml] exit 255 ``` Nothing would notice. The `Watchdog` dead-man's switch exists, but its Alertmanager receiver deliberately has no integrations yet, so it only protects against vmalert death once an external monitor consumes it and alarms on silence. ## Shape `scripts/check-alert-rules.py`, wired into CI on the `ansible` or `scripts` change gates. **Render with the real variables under `StrictUndefined`.** All 34 variables the template uses resolve with no vault password: 30 from the monitoring role's defaults, 4 from plaintext `group_vars/all/vars.yml`, plus `ansible_managed`. So nothing needs faking, and strictness is the point. Substituting a placeholder for an undefined variable renders `> 1`, which parses, so the check would pass on a role default renamed out from under the template. That is the most likely failure of all. **Parse with vmalert, not promtool.** promtool works on today's file but is the wrong engine: vmalert is VictoriaMetrics and accepts MetricsQL that promtool rejects, so promtool could only approximate and would drift toward false failures. `-dryRun` hits the same config parse the real startup does and needs no datasource. The tag is read from the role defaults, so no second literal can drift. PyYAML and Jinja2 both arrive with the `ansible` baked into the runner image (#65), so nothing new is installed. ## The self-test earned its keep immediately `--self-test` mutates the inputs and asserts the check fails. It runs in CI, not just once by hand. On its first run it failed, and it was right to. A threshold set to null renders the literal string `None`, and `x > None` is legal PromQL, because `None` parses as a vector selector. vmalert accepted the file and the rule could never have fired. `StrictUndefined` did not help either, because the key existed. Fixed by dropping null values at the merge step so the strict render rejects them. ``` self-test, proving the check is not vacuous: ok unmodified tree passes ok unbalanced paren in a rule expression is caught ok a role default renamed out from under the template is caught ok a variable emptied to a blank value is caught self-test: 4/4. The check fails when it should ``` ## Verification - `./scripts/check-alert-rules.py` reports `OK: 49 vmalert rules render and parse`, matching the 49 `- alert:` lines in the template. - `./scripts/check-alert-rules.py --self-test` passes 4/4, and each case was observed failing before the corresponding guard existed. - `ruff`, `prettier`, `markdownlint`, `ansible-lint`, `shellcheck` and `gitleaks` all clean. ## Not in scope - The vmalert image is not in `registry_mirror_images`, so this step pulls from docker.io. That leaves it as one of the few CI steps still reaching outside the instance (#137). Mirroring it is a cheap follow-up. - Giving the `Watchdog` heartbeat an external consumer is the larger gap, and is what makes a vmalert failure silent today. Not tracked yet. - This is the static half only. The functional Tier-0 smoke story stays #104 / bitborg-docs#37.

supernaut lade till 1 incheckning

2026-08-18 18:53:40 +00:00

ci: validate the vmalert rule set before it can reach production

ci / ci (pull_request) Successful in 1m52s

Detaljer

201a6fd317

The 49 rules in roles/monitoring/templates/alert-rules.yml.j2 had no gate. The
template task carries no `validate:` and notifies `Restart vmalert` directly, so
a typo went straight to prod.

vmalert parses the rule file as a whole, and the unit does not pass
`-rule.validateExpressions=false`, so one unbalanced paren is fatal to all 49
rules rather than to one. `Restart=always` then makes it a crash loop. Measured
against the pinned v1.147.0: real startup exits 255 at main.go:167. Nothing
would notice, because the Watchdog dead-man's switch is routed to a receiver
with no integrations yet, so it only catches vmalert death once an external
monitor consumes it and alarms on silence.

scripts/check-alert-rules.py renders the template with the real role and group
vars under jinja2.StrictUndefined, then parses the result with vmalert itself at
the tag pinned in the role defaults. Strictness is the point: substituting a
placeholder for an undefined variable renders `> 1`, which parses, so the check
would pass on a role default renamed out from under the template. vmalert rather
than promtool because promtool is the wrong engine, rejecting MetricsQL that
vmalert accepts.

`--self-test` mutates the inputs four ways and asserts the check fails. It runs
in CI, not just once by hand, and it earned that on its first run: it caught
this check accepting a null threshold, which renders the literal `None`, and
`x > None` is legal PromQL, so the rule parsed and could never fire.

Refs #439

supernaut lade till 1 incheckning

2026-08-18 18:57:54 +00:00

docs: record the alert-rule check as the third hook exception

ci / ci (pull_request) Successful in 1m48s

Detaljer

8e58115d4f

README said "Three deliberate differences" and listed two. The new check is
genuinely the third: like the smoke tests it does a container run and an image
pull, so it is CI-only rather than a per-push hook job.

Refs #439

supernaut sammanfogade incheckning 2c0181d2f0 till main

2026-08-18 19:15:37 +00:00

supernaut tog bort grenen feat/validate-alert-rules

2026-08-18 19:15:38 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-18 19:15:38 +00:00

ci: validate the vmalert rule set before it can reach production (#440)

supernaut refererade till denna ändringsförfrågan

2026-08-18 20:11:39 +00:00

observability: the alert that watches the watchdog matches zero series #442

supernaut refererade denna ändringsförfrågan från en incheckning

2026-10-02 17:56:57 +00:00