Incident 2026-07-17..30: sign-up broken for 13 days — Kanidm entry-manager delegation never followed ADR 0029/0036 #271

Nytt ärende

Stängd

öppnade 2026-07-30 22:41:37 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-07-30 22:41:37 +00:00

Ägare

Incident record, filed retroactively. The outage is resolved and every remediation has shipped;
this issue exists so the incident has a searchable record rather than living only in commit
messages. Filed closed.

Impact

Sign-up was broken in production from 2026-07-17 to 2026-07-30 — 13 days. Every new-user
registration failed. No alert fired; the outage surfaced only because someone tried to register and
said so.

Root cause

The portal adds each new sign-up to Kanidm tier groups. Member-write on those groups is delegated to
idm_bitborg_ent_managers via entry-managed-by. That delegation was a manual
kanidm group set-entry-manager step — kanidm-provision (pinned) had no ACP support — and it was
restated in three places that had to agree.

ADR 0029 moved sign-up from tier_basic to tier_participant; ADR 0036 added tier_trial. The
delegation followed neither, so the portal was denied member-write on precisely the groups it
writes:

Group	Was delegated	Portal writes it
`tier_basic`	yes	no — superseded by ADR 0029
`ent_renovate`	yes	yes
`tier_participant`	no	every non-trial sign-up (ADR 0029)
`tier_trial`	no	every trial sign-up (ADR 0036)
`forgejo_users`	no	every sign-up (ADR 0014)

forgejo_users was a second, independent instance of the same gap — found by the drill written
during remediation, not by the original investigation.

Why it went unnoticed for 13 days

Nothing asserted the delegation at apply time, so a clean site.yml run proved nothing about it.
No alert covered sign-up failure.
Unit tests and local e2e structurally could not catch it: the failure is a production
permission/config mismatch rather than logic, and there is no Kanidm in the local harness.
The portal's error message did not name the group or the likely cause, so the 404 read as a
generic upstream fault.

Remediation (all shipped)

Change	Where
Single source of truth (`kanidm_portal_managed_groups`) consumed by state, gate and drill	#262
Apply-time health gate asserting the portal can actually manage each group	#262
Loki alert on sign-up failure	#262
`scripts/signup-drill.sh` + runbook section "Sign-up is broken (`group add -> 404`)"	#262
Health gate wrongly failing the apply on the monitoring host	#263
Delegation made declarative via a carried kanidm-provision patch (`entryManagedBy`)	#268
Portal error messages name the group and the likely cause, plus regression tests	bitborg-web#112

The manual set-entry-manager step no longer exists — #268 removed the class of failure rather than
just this instance of it.

Verification

Delegations restored in production and confirmed; pnpm signup:drill passing; apply-time health
gates green; sign-up verified working end-to-end in production on 2026-07-30.

Follow-up

None outstanding. Retained as evidence for the ISMS incident-management control — see
gitborg/gitborg-docs#30.

**Incident record, filed retroactively.** The outage is resolved and every remediation has shipped; this issue exists so the incident has a searchable record rather than living only in commit messages. Filed closed. ## Impact **Sign-up was broken in production from 2026-07-17 to 2026-07-30 — 13 days.** Every new-user registration failed. No alert fired; the outage surfaced only because someone tried to register and said so. ## Root cause The portal adds each new sign-up to Kanidm tier groups. Member-write on those groups is delegated to `idm_bitborg_ent_managers` via `entry-managed-by`. That delegation was a **manual** `kanidm group set-entry-manager` step — kanidm-provision (pinned) had no ACP support — and it was restated in three places that had to agree. ADR 0029 moved sign-up from `tier_basic` to `tier_participant`; ADR 0036 added `tier_trial`. **The delegation followed neither**, so the portal was denied member-write on precisely the groups it writes: | Group | Was delegated | Portal writes it | | ------------------- | ------------- | ------------------------------------ | | `tier_basic` | yes | no — superseded by ADR 0029 | | `ent_renovate` | yes | yes | | `tier_participant` | **no** | **every non-trial sign-up** (ADR 0029) | | `tier_trial` | **no** | **every trial sign-up** (ADR 0036) | | `forgejo_users` | **no** | **every sign-up** (ADR 0014) | `forgejo_users` was a second, independent instance of the same gap — found by the drill written during remediation, not by the original investigation. ## Why it went unnoticed for 13 days - Nothing asserted the delegation at apply time, so a clean `site.yml` run proved nothing about it. - No alert covered sign-up failure. - Unit tests and local e2e structurally could not catch it: the failure is a production permission/config mismatch rather than logic, and there is no Kanidm in the local harness. - The portal's error message did not name the group or the likely cause, so the 404 read as a generic upstream fault. ## Remediation (all shipped) | Change | Where | | --------------------------------------------------------------------------------------- | ----- | | Single source of truth (`kanidm_portal_managed_groups`) consumed by state, gate and drill | #262 | | Apply-time health gate asserting the portal can actually manage each group | #262 | | Loki alert on sign-up failure | #262 | | `scripts/signup-drill.sh` + runbook section "Sign-up is broken (`group add -> 404`)" | #262 | | Health gate wrongly failing the apply on the monitoring host | #263 | | Delegation made **declarative** via a carried kanidm-provision patch (`entryManagedBy`) | #268 | | Portal error messages name the group and the likely cause, plus regression tests | bitborg-web#112 | The manual `set-entry-manager` step no longer exists — #268 removed the class of failure rather than just this instance of it. ## Verification Delegations restored in production and confirmed; `pnpm signup:drill` passing; apply-time health gates green; sign-up verified working end-to-end in production on 2026-07-30. ## Follow-up None outstanding. Retained as evidence for the ISMS incident-management control — see gitborg/gitborg-docs#30.

supernaut

2026-07-30 22:41:44 +00:00

stängde ärendet
lade till etiketterna
area/identity

area/security

area/web

type
bug

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-30 22:41:45 +00:00

Logga in för att delta i denna konversation.

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra#271

Ingen beskrivning angiven.

Rader
Kolumner