bitborg/bitborg-infra

Bevaka

Avgrena 0

feat(renovate): alert when an update is parked in the dashboard indefinitely #437

Sammanfogat

supernaut sammanfogade 1 incheckning från feat/renovate-dashboard-ageing in i main

2026-08-18 13:19:30 +00:00

supernaut kommenterad

2026-08-18 12:43:44 +00:00

Ägare

Closes the last open acceptance criterion on #430: "If the cause is systemic, a check catches it."

What it catches

Renovate can park an update for ever with no error anywhere. minimumReleaseAgeBehaviour defaults
to timestamp-required, the docker datasource often returns no releaseTimestamp, and a release-age
check that needs a timestamp can never pass without one. Seven image updates — the git server and the
database among them — sat in the Dependency Dashboard's "Pending Status Checks" section for weeks,
each showing an approvePr-branch action as though a branch existed. None was created. Nothing logged
an error at info. The only symptom was an entry that never moved.

scripts/check-renovate-annotations.py cannot see it, and says so in its own docstring's terms: the
annotation is present and correct, so the dependency IS watched. It is watched and then silently
parked. This measures the one signal that separates the two — how long an entry has sat still.

Shape

An exporter runs at the tail of each nightly Renovate run (no new timer, no new role), reads every
dashboard the bot's token can see, and publishes a first-seen timestamp per held update. Three alerts
consume it:

Alert	Guards
`RenovateUpdateHeldTooLong`	the finding: an update parked past 10 days
`RenovateDashboardGuardDegraded`	the guard is partially blind (renamed section, or a repo it could not read)
`RenovateDashboardScanStale`	the guard stopped running at all

The second and third exist because a guard whose failure is invisible is not a guard, and the bug this
replaces was itself invisible.

Decisions worth reviewing

Identity is the Renovate branch, not the entry title. renovate/…-vmagent-1.x stayed stable while
its target moved v1.149.0 → v1.150.0 mid-stall. Keying on the title would have reset the clock on
every upstream release and hidden the stall for ever.

The clock spans all holding sections. Shuffling from "Pending Status Checks" to "Awaiting Schedule"
is not progress, so the age carries over. It resets only when the update leaves every holding section,
i.e. a PR finally exists.

State is pruned only for repos that scanned cleanly. A transient API error must not be read as "the
update progressed" and quietly restart a weeks-old clock.

A first-seen timestamp, not a long for:. vmalert's pending timer resets when vmalert restarts, so
for: 10d might never fire — a guard that cannot fire is worse than none. Same shape as
ForgejoTokenRotationDue.

pending_approval is excluded in the alert, not the exporter. dependencyDashboardApproval gates
majors of the git server and database on purpose (ADR 0023), so one legitimately sits there for ever.
The exporter stays a measurement and the policy lives with the policy.

Unrecognised section headings are counted and alerted on. If upstream renames one, the parser goes
blind and reports zero held updates — which reads as "all healthy". That vacuous pass is the exact
failure mode this whole change exists to prevent, so it is made loud.

Deployed as a plain .py, not a .j2. ruff lints ansible/**/*.py and never a template, and the
logic is unit-testable. Environment-specific values arrive as CLI arguments. Same pattern as
runner-controller/files/controller.py.

Threshold 10 days, tunable via alert_renovate_update_held_days. minimumReleaseAge is 3 days and
the shared preset's window is weekly, so a legitimately waiting update can be ~10 days old. 10 is the
smallest value that cannot false-fire.

Known, accepted weakness: losing the state file resets the clocks. It is derived data, regenerable by
waiting, and RenovateDashboardScanStale covers the case that matters more (the exporter not running).

Verified — every check shown able to fail

21/21 logic tests, wired into CI as its own step. Two are negative controls: a renamed section,
and a clock surviving an unscanned repo. Mutation-tested: treating "Open" as a holding section
fails 2 checks; dropping the unscanned-repo guard fails 1.
Rendered alert-rules.yml.j2 passes promtool check rules (49 rules). An unbalanced paren in the
new expression makes it fail with unclosed left parenthesis.
The exporter's output passes promtool check metrics (rc 0); a malformed sample exits 1.
ruff (pinned 0.16.1) clean, ansible-lint clean on the production profile, --syntax-check clean,
check-metric-names.py consistent across 209 files, check-renovate-annotations.py clean.

The fixture in the test file is a trimmed copy of the real dashboard as it stood on 2026-08-18,
including the seven stalled entries. A synthetic fixture would have been weaker: the bug was invisible
precisely because the dashboard looked healthy, so the test asserts against the shape that fooled us.

Two things for reviewers

Nothing in this repo validated alert rules before. promtool check rules was run by hand here.
Wiring it into CI needs Jinja rendering with resolved role vars, so it is worth its own change rather
than being smuggled into this one.

This needs one apply to take effect (--tags renovate,monitoring). Predicted changes: install the
exporter, create its state dir, re-render the wrapper script, re-render the alert rules and reload
vmalert. No service interruption — the exporter first runs on the next nightly Renovate run, and
publishes nothing until then.

Refs #430

Closes the last open acceptance criterion on #430: *"If the cause is systemic, a check catches it."* ## What it catches Renovate can park an update for ever with **no error anywhere**. `minimumReleaseAgeBehaviour` defaults to `timestamp-required`, the docker datasource often returns no `releaseTimestamp`, and a release-age check that needs a timestamp can never pass without one. Seven image updates — the git server and the database among them — sat in the Dependency Dashboard's "Pending Status Checks" section for weeks, each showing an `approvePr-branch` action as though a branch existed. None was created. Nothing logged an error at `info`. The only symptom was an entry that never moved. `scripts/check-renovate-annotations.py` cannot see it, and says so in its own docstring's terms: the annotation is present and correct, so the dependency IS watched. It is watched and then silently parked. This measures the one signal that separates the two — how long an entry has sat still. ## Shape An exporter runs at the tail of each nightly Renovate run (no new timer, no new role), reads every dashboard the bot's token can see, and publishes a first-seen timestamp per held update. Three alerts consume it: | Alert | Guards | | --- | --- | | `RenovateUpdateHeldTooLong` | the finding: an update parked past 10 days | | `RenovateDashboardGuardDegraded` | the guard is partially blind (renamed section, or a repo it could not read) | | `RenovateDashboardScanStale` | the guard stopped running at all | The second and third exist because a guard whose failure is invisible is not a guard, and the bug this replaces was itself invisible. ## Decisions worth reviewing **Identity is the Renovate branch, not the entry title.** `renovate/…-vmagent-1.x` stayed stable while its target moved v1.149.0 → v1.150.0 *mid-stall*. Keying on the title would have reset the clock on every upstream release and hidden the stall for ever. **The clock spans all holding sections.** Shuffling from "Pending Status Checks" to "Awaiting Schedule" is not progress, so the age carries over. It resets only when the update leaves every holding section, i.e. a PR finally exists. **State is pruned only for repos that scanned cleanly.** A transient API error must not be read as "the update progressed" and quietly restart a weeks-old clock. **A first-seen timestamp, not a long `for:`.** vmalert's pending timer resets when vmalert restarts, so `for: 10d` might never fire — a guard that cannot fire is worse than none. Same shape as `ForgejoTokenRotationDue`. **`pending_approval` is excluded in the alert, not the exporter.** `dependencyDashboardApproval` gates majors of the git server and database on purpose (ADR 0023), so one legitimately sits there for ever. The exporter stays a measurement and the policy lives with the policy. **Unrecognised section headings are counted and alerted on.** If upstream renames one, the parser goes blind and reports zero held updates — which reads as "all healthy". That vacuous pass is the exact failure mode this whole change exists to prevent, so it is made loud. **Deployed as a plain `.py`, not a `.j2`.** ruff lints `ansible/**/*.py` and never a template, and the logic is unit-testable. Environment-specific values arrive as CLI arguments. Same pattern as `runner-controller/files/controller.py`. **Threshold 10 days**, tunable via `alert_renovate_update_held_days`. `minimumReleaseAge` is 3 days and the shared preset's window is weekly, so a legitimately waiting update can be ~10 days old. 10 is the smallest value that cannot false-fire. Known, accepted weakness: losing the state file resets the clocks. It is derived data, regenerable by waiting, and `RenovateDashboardScanStale` covers the case that matters more (the exporter not running). ## Verified — every check shown able to fail - **21/21** logic tests, wired into CI as its own step. Two are negative controls: a renamed section, and a clock surviving an unscanned repo. **Mutation-tested:** treating "Open" as a holding section fails 2 checks; dropping the unscanned-repo guard fails 1. - Rendered `alert-rules.yml.j2` passes `promtool check rules` (49 rules). An unbalanced paren in the new expression makes it fail with `unclosed left parenthesis`. - The exporter's output passes `promtool check metrics` (rc 0); a malformed sample exits 1. - `ruff` (pinned 0.16.1) clean, `ansible-lint` clean on the production profile, `--syntax-check` clean, `check-metric-names.py` consistent across 209 files, `check-renovate-annotations.py` clean. The fixture in the test file is a trimmed copy of the **real** dashboard as it stood on 2026-08-18, including the seven stalled entries. A synthetic fixture would have been weaker: the bug was invisible precisely because the dashboard looked healthy, so the test asserts against the shape that fooled us. ## Two things for reviewers **Nothing in this repo validated alert rules before.** `promtool check rules` was run by hand here. Wiring it into CI needs Jinja rendering with resolved role vars, so it is worth its own change rather than being smuggled into this one. **This needs one apply** to take effect (`--tags renovate,monitoring`). Predicted changes: install the exporter, create its state dir, re-render the wrapper script, re-render the alert rules and reload vmalert. No service interruption — the exporter first runs on the next nightly Renovate run, and publishes nothing until then. Refs #430

supernaut lade till 1 incheckning

2026-08-18 12:43:44 +00:00

feat(renovate): alert when an update is parked in the dashboard indefinitely

ci / ci (pull_request) Successful in 1m46s

Detaljer

a5ca665459

#430's remaining acceptance criterion: a check that catches this class.

Renovate can park an update for ever with no error anywhere. `minimumReleaseAgeBehaviour` defaults
to `timestamp-required`, the docker datasource often returns no `releaseTimestamp`, and a release-age
check that needs a timestamp can never pass without one. Seven image updates — the git server and
the database among them — sat in the Dependency Dashboard's "Pending Status Checks" section for
weeks, each showing an `approvePr-branch` action as though a branch existed. None was created.
Nothing logged an error at `info`. The only symptom was an entry that never moved.

`scripts/check-renovate-annotations.py` cannot see it: the annotation is present and correct, so the
dependency IS watched. It is watched and then silently parked. This measures the one signal that
separates the two — how long an entry has sat still.

An exporter runs at the tail of each nightly Renovate run, reads every dashboard the bot's token can
see, and publishes a first-seen timestamp per held update. Three alerts consume it: the finding
itself, plus two that guard the guard.

Design choices that are load-bearing:

- Identity is the Renovate BRANCH, not the entry title. `renovate/…-vmagent-1.x` stayed stable while
the target moved v1.149.0 → v1.150.0 mid-stall; keying on the title would have reset the clock on
every upstream release and hidden the stall for ever.
- The clock spans all holding sections. Shuffling from "Pending Status Checks" to "Awaiting Schedule"
is not progress, so the age carries; it resets only when a PR finally exists.
- State is pruned only for repos that scanned cleanly, so a transient API error cannot quietly
restart a weeks-old clock.
- A first-seen TIMESTAMP, not a long `for:`. vmalert's pending timer resets on restart, so `for: 10d`
might never fire. Same shape as ForgejoTokenRotationDue instead.
- `pending_approval` is excluded in the alert, not the exporter: dependencyDashboardApproval gates
majors of the git server and database on purpose (ADR 0023), so one sits there legitimately. The
exporter stays a measurement; the policy lives with the policy.
- Unrecognised section headings are counted and alerted on. If upstream renames one, the parser goes
blind and would report zero held updates, which reads as "all healthy" — the vacuous pass this
whole thing exists to prevent.
- Deployed as a plain .py rather than a .j2, because ruff lints `ansible/**/*.py` and never a
template. Environment-specific values arrive as CLI arguments.

Threshold is 10 days: minimumReleaseAge is 3 and the schedule window is weekly, so ~10 is the
smallest value that cannot false-fire.

Verified, each check shown able to fail:

- 21/21 logic tests pass, wired into CI. Two are negative controls (a renamed section, and a clock
reset by an unscanned repo). Mutation-tested: treating "Open" as holding fails 2 checks, dropping
the unscanned-repo guard fails 1.
- Rendered alert-rules.yml.j2 passes `promtool check rules` (49 rules); an unbalanced paren in the
new expression fails it.
- The exporter's output passes `promtool check metrics`; a malformed sample exits 1.
- ruff (pinned 0.16.1), ansible-lint (production profile), --syntax-check, check-metric-names.py and
check-renovate-annotations.py all clean.

Note for reviewers: nothing in this repo validated alert rules before, so `promtool check rules` was
run by hand here. Wiring it into CI needs Jinja rendering with resolved role vars and is worth its
own change.

Refs #430

supernaut sammanfogade incheckning b49484c2f7 till main

2026-08-18 13:19:30 +00:00

supernaut tog bort grenen feat/renovate-dashboard-ageing

2026-08-18 13:19:30 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-18 13:19:31 +00:00

feat(renovate): alert when an update is parked in the dashboard indefinitely (#437)

supernaut refererade till denna ändringsförfrågan

2026-08-18 13:32:50 +00:00

fix(renovate): the dashboard exporter did not know "PR Closed (Blocked)" #438

supernaut refererade denna ändringsförfrågan från en incheckning