Reconciler has no deadman/staleness alert (dead timer → silent fail-open quota) #123

Nytt ärende

Stängd

öppnade 2026-07-19 06:58:01 +00:00 av supernaut · 1 kommentar

supernaut kommenterad

2026-07-19 06:58:01 +00:00

Ägare

Severity: HIGH — pre-onboarding infra audit (2026-07-19). Compounds the fail-open quota issue.

The reconciler writes a last_run_timestamp but nothing alerts on its staleness. ansible/roles/monitoring/templates/alert-rules.yml.j2:156-163 has only ReconcilerFailed: status != 0. A dead timer leaves status at the last-good 0 → no alert fires, while new users sit on the fail-open -1 quota default undetected.

Backups already have BackupStale; the reconciler has no equivalent deadman.

Ask

Add a staleness alert mirroring BackupStale:
time() - gitborg_reconciler_last_run_timestamp_seconds > <threshold> (e.g. 45 min = 3 missed 15-min ticks). Effort S.

**Severity: HIGH** — pre-onboarding infra audit (2026-07-19). Compounds the fail-open quota issue. The reconciler writes a `last_run_timestamp` but nothing alerts on its staleness. `ansible/roles/monitoring/templates/alert-rules.yml.j2:156-163` has only `ReconcilerFailed: status != 0`. A **dead timer** leaves status at the last-good `0` → no alert fires, while new users sit on the fail-open `-1` quota default undetected. Backups already have `BackupStale`; the reconciler has no equivalent deadman. ### Ask Add a staleness alert mirroring `BackupStale`: `time() - gitborg_reconciler_last_run_timestamp_seconds > <threshold>` (e.g. 45 min = 3 missed 15-min ticks). Effort S.

supernaut lade till etiketterna

area/observability

ready-for-implementation

type

task

2026-07-19 06:59:33 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-19 07:04:40 +00:00

supernaut kommenterad

2026-07-19 22:03:40 +00:00

Upphovsperson

Ägare

✅ Resolved — shipped in PR #150 (part of the #120/#121 fail-open work), applied + verified 2026-07-19.

Added exactly the requested deadman: a ReconcilerStale critical alert mirroring BackupStale —
(time() - gitborg_reconciler_last_run_timestamp_seconds) / 60 > {{ alert_reconciler_stale_minutes }} (default 30 min). The reconciler stamps its timestamp metric on every run (success or fail), so this fires when the timer isn't running at all — the dead-timer / silent fail-open case this issue describes.

Verified live: ReconcilerStale present in the deployed vmalert rules on the monitoring VM (loaded cleanly, no parse errors). The reconcile interval was also shortened 15→5 min in the same PR, shrinking the window further. Closing (it was left open only because PR #150 referenced #120/#121, not this issue).

✅ **Resolved — shipped in PR #150** (part of the #120/#121 fail-open work), applied + verified 2026-07-19. Added exactly the requested deadman: a `ReconcilerStale` critical alert mirroring `BackupStale` — `(time() - gitborg_reconciler_last_run_timestamp_seconds) / 60 > {{ alert_reconciler_stale_minutes }}` (default 30 min). The reconciler stamps its timestamp metric on every run (success or fail), so this fires when the timer isn't running at all — the dead-timer / silent fail-open case this issue describes. Verified live: `ReconcilerStale` present in the deployed vmalert rules on the monitoring VM (loaded cleanly, no parse errors). The reconcile interval was also shortened 15→5 min in the same PR, shrinking the window further. Closing (it was left open only because PR #150 referenced #120/#121, not this issue).

supernaut stängde ärendet

2026-07-19 22:03:41 +00:00

Logga in för att delta i denna konversation.

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra#123

Ingen beskrivning angiven.

Rader
Kolumner