feat(monitoring): alert on identity projection failures and refused account creation #347

Sammanfogat

supernaut sammanfogade 1 incheckning från feat/identity-projection-alerts in i main

2026-08-02 19:24:05 +00:00

supernaut kommenterad

2026-08-02 15:56:14 +00:00

Ägare

Closes #326. Part of the identity-validation epic.

Important

IdentityProjectionDegraded is inert until the reconciler is released and its image pin bumped.
The two gauges it keys on ship in the reconciler's next version; group_vars still pins the
current one. Order is: reconciler release → image pin bump → apply monitoring. No tag was invented
here.

Two identity failure modes were unalertable.

1. Projection failures were reported under the wrong name. A refused identity write raised the
Actions enforcement gauge, so the alert that fired said the reconciler was not applying CI tier
enforcement and sent the operator after a token scope. The reconciler now publishes its own signal;
this adds the rule that keys on it.

Three properties of that metric shaped the rule, and each is easy to get wrong:

It is a per-run gauge, not a cumulative counter — hence no _total suffix. The condition keys
on the degraded gauge alone and the count appears only in the annotation, labelled as such. No
rate() or increase() anywhere. The idiom that would carry the count into $value was
deliberately not used: it makes the alert silently unfirable if the two series ever diverge, and
firing reliability beats a tidier summary.
It never self-clears — the projection recomputes the same diff every run — so for: is
generous, matching its sibling, and the annotation tells the operator to fix the value, not to
retry or restart.
A fatal run zeroes both flags. The rule therefore also requires a clean last run, so its claim
is true by construction and it stays quiet during an unrelated outage, which the deadman and status
alerts already own. The vector match was verified against production using the existing equivalent
pair before committing to it. The residual cost — the window restarts after an outage, delaying
re-fire on a sticky condition — is stated in the file comment.

2. Refused account creation at first sign-in was watched by nothing. If a name is valid at the
identity provider but reserved on the git host, creation fails as a 500: the portal is not in the
request, the reconciler cannot see an account that was never created, and no metric moves. Adds a
log-based rule; one occurrence is one locked-out user.

Production is clean — 14 days and 1.9 M lines scanned, no match — so the matcher is derived from
upstream 16.0.1 by tracing the actual path: the OAuth callback creates users with an empty template
name, which routes every creation error to the server-error branch, which logs at [E] with the
handler name. The account-linking handler is included too, because linking is enabled, so a collision
fails the person out identically on the same request.

Verification

Both rules validated against rendered output, not the template. vmalert -dryRun at the pinned
production version reports 42 rules, exit 0; promtool check rules agrees and additionally validates
the annotation templates. For the log rule, the real ruler was run in a throwaway container at the
pinned version — it loaded the group and evaluated the query, returning 200.

Prettier and markdownlint clean.

Closes #326. Part of the identity-validation epic. > [!IMPORTANT] > **`IdentityProjectionDegraded` is inert until the reconciler is released and its image pin bumped.** > The two gauges it keys on ship in the reconciler's next version; `group_vars` still pins the > current one. Order is: reconciler release → image pin bump → apply monitoring. No tag was invented > here. Two identity failure modes were unalertable. **1. Projection failures were reported under the wrong name.** A refused identity write raised the *Actions enforcement* gauge, so the alert that fired said the reconciler was not applying CI tier enforcement and sent the operator after a token scope. The reconciler now publishes its own signal; this adds the rule that keys on it. Three properties of that metric shaped the rule, and each is easy to get wrong: - It is a **per-run gauge, not a cumulative counter** — hence no `_total` suffix. The condition keys on the degraded gauge alone and the count appears only in the annotation, labelled as such. No `rate()` or `increase()` anywhere. The idiom that would carry the count into `$value` was deliberately **not** used: it makes the alert silently unfirable if the two series ever diverge, and firing reliability beats a tidier summary. - **It never self-clears** — the projection recomputes the same diff every run — so `for:` is generous, matching its sibling, and the annotation tells the operator to fix the *value*, not to retry or restart. - **A fatal run zeroes both flags.** The rule therefore also requires a clean last run, so its claim is true by construction and it stays quiet during an unrelated outage, which the deadman and status alerts already own. The vector match was verified against production using the existing equivalent pair before committing to it. The residual cost — the window restarts after an outage, delaying re-fire on a sticky condition — is stated in the file comment. **2. Refused account creation at first sign-in was watched by nothing.** If a name is valid at the identity provider but reserved on the git host, creation fails as a 500: the portal is not in the request, the reconciler cannot see an account that was never created, and no metric moves. Adds a log-based rule; one occurrence is one locked-out user. Production is clean — 14 days and 1.9 M lines scanned, no match — so the matcher is derived from upstream 16.0.1 by tracing the actual path: the OAuth callback creates users with an empty template name, which routes every creation error to the server-error branch, which logs at `[E]` with the handler name. The account-linking handler is included too, because linking is enabled, so a collision fails the person out identically on the same request. ### Verification Both rules validated against **rendered** output, not the template. `vmalert -dryRun` at the pinned production version reports 42 rules, exit 0; `promtool check rules` agrees and additionally validates the annotation templates. For the log rule, the real ruler was run in a throwaway container at the pinned version — it loaded the group and evaluated the query, returning 200. Prettier and markdownlint clean.

supernaut lade till 1 incheckning

2026-08-02 15:56:15 +00:00

feat(monitoring): alert on identity projection failures and refused account creation

ci / ci (pull_request) Successful in 1m47s

Detaljer

57420f94bb

Two identity failure modes had no alert of their own.

IdentityProjectionDegraded keys on the reconciler's own identity gauge instead of
borrowing the Actions-enforcement one, so a refused identity write no longer fires
an alert that names the wrong subsystem and sends the operator after a token scope
that was never the problem. Gated on last_run_status == 0: a fatal run zeroes the
identity gauges, and that case is already ReconcilerFailed's and ReconcilerStale's,
so the identity alert stays quiet through an unrelated outage instead of
half-firing on a zeroed gauge. for: 1h (~12 consecutive refused runs) because the
flag never self-clears — the projection recomputes the same diff every run, so the
fix is to change the value in Kanidm or free the colliding Forgejo account, never
to retry. identity_failures is a per-run gauge, not a counter, so it is quoted for
triage rather than rated.

ForgejoAccountCreationRefused (Loki ruler) watches the failure nothing could see:
a username valid at the identity provider but reserved on the git host turns first
sign-in into a 500, the portal is not in the request, the reconciler cannot see an
account that was never created, and no metric moves. Matcher derived from Forgejo
16.0.1 — the OIDC callback creates the user with an empty template name, so every
CreateUser error takes the ctx.ServerError branch and logs "[E] CreateUser: <err>".
Production has no occurrence in 14 days, so there was no live sample to copy.

Validated on the rendered templates, not the Jinja: vmalert -dryRun and promtool
check rules for the metric file, and a throwaway Loki 3.7.3 ruler that loaded and
evaluated the log rule.

Closes #326.

supernaut tvångsskickade feat/identity-projection-alerts från 57420f94bb

ci / ci (pull_request) Successful in 1m47s

Detaljer

till 4a0d239233

ci / ci (pull_request) Successful in 2m2s

Detaljer

2026-08-02 18:30:14 +00:00

Jämför

supernaut lade till 1 incheckning

2026-08-02 19:09:27 +00:00

Merge remote-tracking branch 'origin/main' into feat/identity-projection-alerts

ci / ci (pull_request) Successful in 1m32s