ci: nothing validates the 49 vmalert rules before they reach production #439

Nytt ärende

Stängd

öppnade 2026-08-18 16:14:33 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-18 16:14:33 +00:00

Ägare

The gap

ansible/roles/monitoring/templates/alert-rules.yml.j2 renders 49 alert rules. Nothing checks that
they parse. grep -rn promtool over the whole repo returns nothing, and the Render alert rules
task carries no validate:.

The deploy path is that template task, then notify: Restart vmalert. There is no gate anywhere
between a typo and production.

Blast radius

vmalert parses and validates the rule file at startup, and the unit does not pass
-rule.validateExpressions=false. A parse error is fatal for the whole file, not for the one rule.
Measured against the pinned vmalert_image_tag (v1.147.0), with a single extra ( in the
DiskUsageWarning expression:

# real startup, no -dryRun
fatal  app/vmalert/main.go:167  cannot parse configuration file: failed to parse [rules.yml]
exit 255

# -dryRun exercises the same parse
fatal  app/vmalert/main.go:112  failed to parse "[rules.yml]"
exit 255

So one unbalanced paren stops all 49 rules. vmalert.container.j2 sets Restart=always, so the
result is a crash loop rather than a stopped unit.

Nothing currently notices. The Watchdog dead-man's switch exists (vector(1), severity none),
but its Alertmanager receiver deliberately has no integrations yet: it only protects against vmalert
death once an external monitor consumes it and alarms on silence. Until that lands, vmalert dying is
silent, and so is the loss of every other alert.

Proposed gate

A step in .forgejo/workflows/ci.yml gated on steps.changes.outputs.ansible, alongside the
existing python checks. Two stages, because they catch different failures.

Stage 1: render with real variables, under `StrictUndefined`

Every variable the template uses resolves with no vault password:

30 from roles/monitoring/defaults/main.yml
4 from plaintext group_vars/all/vars.yml (bitborg_domain, web_healthcheck_path,
forgejo_oidc_source_name, backup_retention_days)
ansible_managed, which can be any string

So nothing needs faking. Use jinja2.StrictUndefined and the real values.

Do not substitute a placeholder for undefined variables. A placeholder renders > 1, which
parses, so the guard would pass on precisely the bug most likely to occur. A role default renamed
out from under the template is the realistic failure here, and only strict mode catches it.

PyYAML and Jinja2 both ship with the ansible already baked into the ci runner image (#65), so
this installs nothing new. Invoke it with ansible's own interpreter, not a bare python3.

Stage 2: validate with vmalert, not promtool

promtool check rules does work on the rendered file today. It is still the wrong engine. vmalert
is VictoriaMetrics: it accepts MetricsQL and vmalert-specific fields that promtool rejects, so
promtool can only ever approximate, and it drifts toward false failures as rules use more MetricsQL.
vmalert -dryRun at the pinned tag is the exact consumer, needs no datasource, and returns 255 on
failure.

Negative controls

Proven locally, with real exit codes:

Mutation	Expected	Observed
unmodified template	pass	render 0, vmalert 0, "49 rules found"
extra `(` in `DiskUsageWarning`	fail	vmalert 255 (promtool 1)
`alert_disk_warn_pct` renamed in role defaults	fail	render 1, `jinja2 UndefinedError`

Wire these into the check itself, the way test_renovate_dashboard_exporter.py carries its own
negative controls. A validator nobody has watched fail is not evidence.

Notes

Read the vmalert tag from roles/monitoring/defaults/main.yml. Do not pin a second literal in CI:
a duplicated literal drifts, and the drift is invisible.
--user 0 is needed when the rendered file sits on a bind mount the container user cannot read.
Beware cmd | tail in the CI step. $? then reports tail, and the gate is green forever.
This is the static half only. The functional Tier-0 smoke story is #104 / bitborg-docs#37.
Giving the Watchdog heartbeat an external consumer is a separate and larger gap. It is what makes
this one silent, and it does not appear to be tracked yet.

Acceptance

CI fails on an unparseable rule expression.
CI fails when a variable the template uses is undefined.
CI passes on main unchanged, reporting 49 rules.
The negative controls run in CI, not just once by hand.

## The gap `ansible/roles/monitoring/templates/alert-rules.yml.j2` renders 49 alert rules. Nothing checks that they parse. `grep -rn promtool` over the whole repo returns nothing, and the `Render alert rules` task carries no `validate:`. The deploy path is that template task, then `notify: Restart vmalert`. There is no gate anywhere between a typo and production. ## Blast radius vmalert parses and validates the rule file at startup, and the unit does not pass `-rule.validateExpressions=false`. A parse error is fatal for the whole file, not for the one rule. Measured against the pinned `vmalert_image_tag` (v1.147.0), with a single extra `(` in the `DiskUsageWarning` expression: ``` # real startup, no -dryRun fatal app/vmalert/main.go:167 cannot parse configuration file: failed to parse [rules.yml] exit 255 # -dryRun exercises the same parse fatal app/vmalert/main.go:112 failed to parse "[rules.yml]" exit 255 ``` So one unbalanced paren stops all 49 rules. `vmalert.container.j2` sets `Restart=always`, so the result is a crash loop rather than a stopped unit. Nothing currently notices. The `Watchdog` dead-man's switch exists (`vector(1)`, severity `none`), but its Alertmanager receiver deliberately has no integrations yet: it only protects against vmalert death once an external monitor consumes it and alarms on silence. Until that lands, vmalert dying is silent, and so is the loss of every other alert. ## Proposed gate A step in `.forgejo/workflows/ci.yml` gated on `steps.changes.outputs.ansible`, alongside the existing python checks. Two stages, because they catch different failures. ### Stage 1: render with real variables, under `StrictUndefined` Every variable the template uses resolves with no vault password: - 30 from `roles/monitoring/defaults/main.yml` - 4 from plaintext `group_vars/all/vars.yml` (`bitborg_domain`, `web_healthcheck_path`, `forgejo_oidc_source_name`, `backup_retention_days`) - `ansible_managed`, which can be any string So nothing needs faking. Use `jinja2.StrictUndefined` and the real values. Do **not** substitute a placeholder for undefined variables. A placeholder renders `> 1`, which parses, so the guard would pass on precisely the bug most likely to occur. A role default renamed out from under the template is the realistic failure here, and only strict mode catches it. PyYAML and Jinja2 both ship with the `ansible` already baked into the `ci` runner image (#65), so this installs nothing new. Invoke it with ansible's own interpreter, not a bare `python3`. ### Stage 2: validate with vmalert, not promtool `promtool check rules` does work on the rendered file today. It is still the wrong engine. vmalert is VictoriaMetrics: it accepts MetricsQL and vmalert-specific fields that promtool rejects, so promtool can only ever approximate, and it drifts toward false failures as rules use more MetricsQL. `vmalert -dryRun` at the pinned tag is the exact consumer, needs no datasource, and returns 255 on failure. ## Negative controls Proven locally, with real exit codes: | Mutation | Expected | Observed | | --- | --- | --- | | unmodified template | pass | render 0, vmalert 0, "49 rules found" | | extra `(` in `DiskUsageWarning` | fail | vmalert 255 (promtool 1) | | `alert_disk_warn_pct` renamed in role defaults | fail | render 1, `jinja2 UndefinedError` | Wire these into the check itself, the way `test_renovate_dashboard_exporter.py` carries its own negative controls. A validator nobody has watched fail is not evidence. ## Notes - Read the vmalert tag from `roles/monitoring/defaults/main.yml`. Do not pin a second literal in CI: a duplicated literal drifts, and the drift is invisible. - `--user 0` is needed when the rendered file sits on a bind mount the container user cannot read. - Beware `cmd | tail` in the CI step. `$?` then reports `tail`, and the gate is green forever. - This is the static half only. The functional Tier-0 smoke story is #104 / bitborg-docs#37. - Giving the `Watchdog` heartbeat an external consumer is a separate and larger gap. It is what makes this one silent, and it does not appear to be tracked yet. ## Acceptance - [ ] CI fails on an unparseable rule expression. - [ ] CI fails when a variable the template uses is undefined. - [ ] CI passes on `main` unchanged, reporting 49 rules. - [ ] The negative controls run in CI, not just once by hand.

supernaut lade till etiketterna

area/ci

area/observability

ready-for-implementation

type

task

2026-08-18 16:14:44 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-18 16:15:10 +00:00

supernaut refererade till detta ärende från en incheckning

2026-08-18 18:53:07 +00:00

ci: validate the vmalert rule set before it can reach production