monitoring: alert on billing reconcile manual-review lines #510

Nytt ärende

Stängd

öppnade 2026-10-01 18:31:50 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-10-01 18:31:50 +00:00

Ägare

Problem

The billing service logs [reconcile] <id> is paid at the provider with no settled local charge; needs manual review when a paid provider payment cannot be matched or repaired. Nothing alerts on it. The line is logged once per payment, so a missed alert means a paying customer without access, or a charge with no record.

Fix

Add a Loki alert rule on that line for the billing service container. Route it to the normal operator channel. Prove it with a negative control and a synthetic line, as with the other log alerts.

Done when

A matching line in Loki fires the alert within one evaluation interval. The runbook "Billing service" section names the alert and the manual review step.

## Problem The billing service logs `[reconcile] <id> is paid at the provider with no settled local charge; needs manual review` when a paid provider payment cannot be matched or repaired. Nothing alerts on it. The line is logged once per payment, so a missed alert means a paying customer without access, or a charge with no record. ## Fix Add a Loki alert rule on that line for the billing service container. Route it to the normal operator channel. Prove it with a negative control and a synthetic line, as with the other log alerts. ## Done when A matching line in Loki fires the alert within one evaluation interval. The runbook "Billing service" section names the alert and the manual review step.

supernaut lade till etiketterna

ready-for-implementation

type

task

area/observability

2026-10-01 18:38:17 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-10-01 18:57:48 +00:00

supernaut refererade till detta ärende

2026-10-01 20:47:20 +00:00

Payments: deploy the billing service #496

supernaut refererade till detta ärende från en incheckning