Alert when a first-party owner is not in its expected quota group #423

Nytt ärende

Öppen

öppnade 2026-08-16 11:24:49 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-16 11:24:49 +00:00

Ägare

What

Nothing alerts when a first-party owner loses its expected quota group.

Why

The reconciler logs its decision at INFO, in one shape for every org:

[reconciler] org Bitborg: quota=org-default actions=false

That line is the normal, correct output for a tenant org. It is also the output when the
platform's own org has just been demoted to the tenant cap with Actions switched off. Nothing
in the line distinguishes the routine case from the outage, so nothing fires.

On 2026-08-04 that meant a demotion went unnoticed until pushes started failing, and it stayed
broken for ~90 minutes. The reconciler is authoritative and re-asserts its decision every tick, so
the window between "wrong state" and "someone notices" is the entire outage.

The case-sensitivity defect that caused that specific demotion is fixed in
bitborg-auth-reconciler#42. This issue is the detection gap, which is worth closing regardless —
the fixed bug is one of several ways an owner can end up in the wrong group. A bad exempt-list
edit, a group deleted by hand in the Forgejo admin UI, or a future projection change would all land
the same way and be equally invisible.

What to do

Alert when a declared first-party owner is not in its expected quota group. The declarations
already exist as reconciler_user_exempt and reconciler_org_exempt in
roles/reconciler/defaults/main.yml, so the expected state is known without new configuration.

Two candidate shapes, either is fine:

Have the reconciler emit a gauge per first-party owner (in the expected group / not), and alert on
it. Cheapest to reason about, and it reuses the existing bitborg_* metric pipeline.
Or log the first-party demotion at WARN with a distinct message and alert on that in Loki. Less
precise, but needs no metric surface.

Prefer the metric: the log route re-creates the same "is this line normal or not" ambiguity that
caused the gap.

Acceptance

A first-party owner in the wrong quota group raises an alert within a few reconcile ticks.
The alert names the owner and the group it is in.
A tenant org sitting in the tenant group raises nothing.

## What Nothing alerts when a first-party owner loses its expected quota group. ## Why The reconciler logs its decision at INFO, in one shape for every org: ``` [reconciler] org Bitborg: quota=org-default actions=false ``` That line is the normal, correct output for a tenant org. It is also the output when the **platform's own org** has just been demoted to the tenant cap with Actions switched off. Nothing in the line distinguishes the routine case from the outage, so nothing fires. On 2026-08-04 that meant a demotion went unnoticed until pushes started failing, and it stayed broken for ~90 minutes. The reconciler is authoritative and re-asserts its decision every tick, so the window between "wrong state" and "someone notices" is the entire outage. The case-sensitivity defect that caused that specific demotion is fixed in bitborg-auth-reconciler#42. This issue is the detection gap, which is worth closing regardless — the fixed bug is one of several ways an owner can end up in the wrong group. A bad exempt-list edit, a group deleted by hand in the Forgejo admin UI, or a future projection change would all land the same way and be equally invisible. ## What to do Alert when a declared first-party owner is not in its expected quota group. The declarations already exist as `reconciler_user_exempt` and `reconciler_org_exempt` in `roles/reconciler/defaults/main.yml`, so the expected state is known without new configuration. Two candidate shapes, either is fine: - Have the reconciler emit a gauge per first-party owner (in the expected group / not), and alert on it. Cheapest to reason about, and it reuses the existing `bitborg_*` metric pipeline. - Or log the first-party demotion at WARN with a distinct message and alert on that in Loki. Less precise, but needs no metric surface. Prefer the metric: the log route re-creates the same "is this line normal or not" ambiguity that caused the gap. ## Acceptance - A first-party owner in the wrong quota group raises an alert within a few reconcile ticks. - The alert names the owner and the group it is in. - A tenant org sitting in the tenant group raises nothing.

supernaut lade till etiketterna

area/identity

area/observability

ready-for-implementation

type

task

2026-08-16 11:24:56 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-16 11:24:57 +00:00

Logga in för att delta i denna konversation.