feat(monitoring): alert on billing reconcile manual-review line #512

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/510-reconcile-alert in i main

2026-10-01 21:23:14 +00:00

supernaut kommenterad

2026-10-01 21:05:27 +00:00

Ägare

What

New Loki ruler alert BillingReconcileManualReview (warning), same path to Alertmanager as the sign-up and identity log alerts:

sum(count_over_time({container="bitborg-billing"} |~ `\[reconcile\].*needs manual review` [1h])) > 0

for: 0m, so one line fires it. It matches both billing service lines: the once-per-payment line when an orphan is first found, and a per-sweep summary while any orphan is unresolved. The alert stays firing until every orphan is resolved and clears about 1 h later.

Runbook "Billing service" gets a "Manual review alert" subsection with the review step.

Verification

container label comes from Alloy's journald CONTAINER_NAME mapping. bitborg-billing is billing_container_name.
Billing is disabled, so no bitborg-billing stream exists yet. Selector shape proven on bitborg-web: same-shape query with a line filter returned 1, without it 7. Negative control (this pattern on bitborg-web, 24 h): no data.
Template renders and parses (4 groups). check-alert-rules.py covers only the vmalert rules.

Needs

Apply the monitoring tag after merge.
Synthetic-line test once billing is enabled: step added to the #496 enable checklist.

Closes #510

## What New Loki ruler alert `BillingReconcileManualReview` (warning), same path to Alertmanager as the sign-up and identity log alerts: ``` sum(count_over_time({container="bitborg-billing"} |~ `\[reconcile\].*needs manual review` [1h])) > 0 ``` `for: 0m`, so one line fires it. It matches both billing service lines: the once-per-payment line when an orphan is first found, and a per-sweep summary while any orphan is unresolved. The alert stays firing until every orphan is resolved and clears about 1 h later. Runbook "Billing service" gets a "Manual review alert" subsection with the review step. ## Verification - `container` label comes from Alloy's journald `CONTAINER_NAME` mapping. `bitborg-billing` is `billing_container_name`. - Billing is disabled, so no `bitborg-billing` stream exists yet. Selector shape proven on `bitborg-web`: same-shape query with a line filter returned 1, without it 7. Negative control (this pattern on `bitborg-web`, 24 h): no data. - Template renders and parses (4 groups). `check-alert-rules.py` covers only the vmalert rules. ## Needs - Apply the `monitoring` tag after merge. - Synthetic-line test once billing is enabled: step added to the #496 enable checklist. Closes #510

supernaut lade till 2 incheckningar

2026-10-01 21:05:27 +00:00

feat(monitoring): alert on billing reconcile manual-review line 8f50fe5591