monitoring: scrape Alertmanager and add inhibition rules #461

Nytt ärende

Öppen

öppnade 2026-09-08 23:04:26 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-09-08 23:04:26 +00:00

Ägare

AlertmanagerNotificationsFailing watches vmalert_alerts_send_errors_total only. Alertmanager's own alertmanager_notifications_failed_total is not scraped, so an expired ntfy token drops every notification silently.

alertmanager.yml.j2 has no inhibit_rules, so one incident on the services host produces one alert per dependent service.

Add an Alertmanager scrape job and a rule on failed notifications, and inhibition rules: host down inhibits per-service alerts on that host; backup volume unmounted inhibits backup staleness.

`AlertmanagerNotificationsFailing` watches `vmalert_alerts_send_errors_total` only. Alertmanager's own `alertmanager_notifications_failed_total` is not scraped, so an expired ntfy token drops every notification silently. `alertmanager.yml.j2` has no `inhibit_rules`, so one incident on the services host produces one alert per dependent service. Add an Alertmanager scrape job and a rule on failed notifications, and inhibition rules: host down inhibits per-service alerts on that host; backup volume unmounted inhibits backup staleness.

supernaut lade till detta till projektet Bitborg Infrastructure

2026-09-08 23:04:50 +00:00

supernaut lade till etiketterna

2026-09-08 23:05:24 +00:00

Logga in för att delta i denna konversation.