monitoring: alert when Renovate has not run, not just when it fails #330

Nytt ärende

Öppen

öppnade 2026-08-02 12:30:46 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-02 12:30:46 +00:00

Ägare

RenovateRunFailed only fires on gitborg_renovate_last_run_status != 0. That metric is a
node_exporter textfile metric, so it is sticky: it keeps serving the last value written by the
last run, forever. If the timer is masked, the unit is removed, or the wrapper never reaches its
metric write, the last status 0 sits there indefinitely and nothing alerts. A Renovate that has
simply stopped is currently invisible.

The wrapper already writes gitborg_renovate_last_run_timestamp_seconds — it is scraped and unused.

Proposed rule, alongside RenovateRunFailed in
ansible/roles/monitoring/templates/alert-rules.yml.j2:

- alert: RenovateStale
  expr: time() - gitborg_renovate_last_run_timestamp_seconds > 172800
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Renovate has not completed a run in over 48h"
    description: "gitborg_renovate_last_run_timestamp_seconds is more than 48h old. The timer runs daily at 00:30, so two missed nights means the timer, the unit or the wrapper's metric write has stopped. Note this metric is sticky — a stale timestamp is the ONLY signal that Renovate stopped, since last_run_status keeps reporting the last successful value."

48h (not 24h) so a single skipped or long-delayed night does not page; RandomizedDelaySec=300 plus
Persistent=true make the exact fire time jittery.

Acceptance: rule renders, promtool/vmalert accepts it, and it fires in a check by temporarily
back-dating the textfile in a scratch copy (not on the host).

`RenovateRunFailed` only fires on `gitborg_renovate_last_run_status != 0`. That metric is a node_exporter **textfile** metric, so it is sticky: it keeps serving the last value written by the last run, forever. If the timer is masked, the unit is removed, or the wrapper never reaches its metric write, the last `status 0` sits there indefinitely and nothing alerts. A Renovate that has simply stopped is currently invisible. The wrapper already writes `gitborg_renovate_last_run_timestamp_seconds` — it is scraped and unused. **Proposed rule**, alongside `RenovateRunFailed` in `ansible/roles/monitoring/templates/alert-rules.yml.j2`: ```yaml - alert: RenovateStale expr: time() - gitborg_renovate_last_run_timestamp_seconds > 172800 for: 30m labels: severity: warning annotations: summary: "Renovate has not completed a run in over 48h" description: "gitborg_renovate_last_run_timestamp_seconds is more than 48h old. The timer runs daily at 00:30, so two missed nights means the timer, the unit or the wrapper's metric write has stopped. Note this metric is sticky — a stale timestamp is the ONLY signal that Renovate stopped, since last_run_status keeps reporting the last successful value." ``` 48h (not 24h) so a single skipped or long-delayed night does not page; `RandomizedDelaySec=300` plus `Persistent=true` make the exact fire time jittery. **Acceptance:** rule renders, `promtool`/vmalert accepts it, and it fires in a check by temporarily back-dating the textfile in a scratch copy (not on the host).

supernaut lade till etiketterna

area/observability

type

task

2026-08-02 12:31:41 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-02 12:34:52 +00:00

supernaut lade till etiketten

ready-for-implementation

2026-08-11 09:09:22 +00:00

Logga in för att delta i denna konversation.