feat(kanidm): make the entry-manager delegation declarative via a carried patch #268

Sammanfogat

supernaut sammanfogade 2 incheckningar från feat/kanidm-declarative-entry-manager in i main 2026-07-30 21:15:02 +00:00

Konversation 0 Incheckningar 2 Ändrade filer 8 +150 -2218

supernaut kommenterad 2026-07-30 20:54:38 +00:00

Ägare

Kopiera länk

Retires the manual kanidm group set-entry-manager step whose omission broke sign-up for 13 days
(2026-07-17..30). ADR 0029 moved the tier group and ADR 0036 added the trial group; the manual
delegation followed neither, and Kanidm reports a denied member-write as 404, so the symptom was a
bare group add -> 404.

The premise that blocked this was wrong

"kanidm-provision has no ACP support" is true but irrelevant. bitborg does not need access-control
profiles. kanidm group set-entry-manager is one ordinary attribute write:

PUT /v1/group/<group>/_attr/entry_managed_by   ["<manager>"]

That is exactly the request update_entity_attrs() already issues, as the idm_admin it already
authenticates as. The blocker was a missing optional field on struct Group — not an architecture.

The patch (three parts, on the pinned upstream ref)

• state.rs — entry_managed_by: Option<String> on Group (camelCase ⇒ entryManagedBy).
  Omitted/null means leave alone.
• client.rs — extend the SPN-stripping that already exists for member to entry_managed_by.
  Kanidm returns it as name@domain, so without this the comparison never matches what we wrote and
  the attribute is rewritten on every run. kaniop independently normalises the same attribute the
  same way — useful corroboration.
• main.rs — write it in the Syncing group members phase, not in sync_groups(), because
  state.groups is an unordered HashMap: by that phase every group exists, so the manager can safely
  be another provisioned group.

The if let Some(...) guard is load-bearing. update_entity_attrs() treats an empty Vec with
append: false as the values.is_empty() case and issues a DELETE — so passing an empty list for
an undeclared field would strip an entry manager set outside this tool. Omission must mean "leave
alone", never "remove".

Two structural fixes alongside

Decoupled the image tag from the upstream ref. kanidm_provision_image_tag doubled as the git ref
to clone, so a patched image could not be labelled distinctly without breaking the clone. Now
kanidm_provision_upstream_ref (what to clone) and kanidm_provision_image_tag
(v1.3.0-bitborg1, bumped when the patch changes — the marker from #266 catches Containerfile edits,
not patch edits).

Collapsed three copies of the delegated-group list into one — kanidm_portal_managed_groups in
group_vars/all, consumed by the state template and by site.yml's health gate. Three copies that
had to agree is the root shape of the outage. (signup-drill.sh reads the same underlying
web_kanidm_*_group vars, because a shell sed cannot resolve Jinja.)

Verification — each step, not by reasoning

Check	Result
git apply --check against a fresh v1.3.0 clone	clean
cargo build --release on rust:1-trixie	succeeds
field genuinely parsed, not ignored	entryManagedBy: 12345 → invalid type: integer 12345, expected a string
build through the real Containerfile (exercises git apply)	succeeds
patched binary parses a realistic state with entryManagedBy	yes (fails only at the bogus URL)
render the state template offline	valid JSON; entryManagedBy on exactly the 4 managed groups; no key on the other 12
--syntax-check, ansible-lint	clean at production

That wrong-type test is worth noting: it is precisely the test that would have caught the dead
service-accounts block (#267). A known field validates its type; an unknown one is silently
dropped.

git apply rather than patch, deliberately — a context drift on a future ref bump fails the build
loudly instead of half-applying.

What stays

The health gate and signup-drill.sh remain. They become belt-and-braces rather than the only
defence — and the gate is what will catch it if this patch is ever dropped on a ref bump.

The manual command is kept in the runbook as documented break-glass.

Apply note

This bumps the image tag, so the next apply rebuilds the image on the host (native Rust compile,
~3 min on prod based on the last one). Being offered upstream separately; there is no existing issue or
PR for the feature.

Retires the manual `kanidm group set-entry-manager` step whose omission **broke sign-up for 13 days** (2026-07-17..30). ADR 0029 moved the tier group and ADR 0036 added the trial group; the manual delegation followed neither, and Kanidm reports a denied member-write as **404**, so the symptom was a bare `group add -> 404`. ## The premise that blocked this was wrong "kanidm-provision has no ACP support" is true but **irrelevant**. bitborg does not need access-control *profiles*. `kanidm group set-entry-manager` is one ordinary attribute write: ``` PUT /v1/group/<group>/_attr/entry_managed_by ["<manager>"] ``` That is exactly the request `update_entity_attrs()` already issues, as the `idm_admin` it already authenticates as. The blocker was a missing optional field on `struct Group` — not an architecture. ## The patch (three parts, on the pinned upstream ref) - **`state.rs`** — `entry_managed_by: Option<String>` on `Group` (camelCase ⇒ `entryManagedBy`). Omitted/null means **leave alone**. - **`client.rs`** — extend the SPN-stripping that already exists for `member` to `entry_managed_by`. Kanidm returns it as `name@domain`, so without this the comparison never matches what we wrote and the attribute is rewritten on every run. `kaniop` independently normalises the same attribute the same way — useful corroboration. - **`main.rs`** — write it in the *Syncing group members* phase, **not** in `sync_groups()`, because `state.groups` is an unordered `HashMap`: by that phase every group exists, so the manager can safely be another provisioned group. **The `if let Some(...)` guard is load-bearing.** `update_entity_attrs()` treats an empty `Vec` with `append: false` as the `values.is_empty()` case and issues a **DELETE** — so passing an empty list for an undeclared field would *strip* an entry manager set outside this tool. Omission must mean "leave alone", never "remove". ## Two structural fixes alongside **Decoupled the image tag from the upstream ref.** `kanidm_provision_image_tag` doubled as the git ref to clone, so a patched image could not be labelled distinctly without breaking the clone. Now `kanidm_provision_upstream_ref` (what to clone) and `kanidm_provision_image_tag` (`v1.3.0-bitborg1`, bumped when the patch changes — the marker from #266 catches Containerfile edits, not patch edits). **Collapsed three copies of the delegated-group list into one** — `kanidm_portal_managed_groups` in `group_vars/all`, consumed by the state template *and* by site.yml's health gate. **Three copies that had to agree is the root shape of the outage.** (`signup-drill.sh` reads the same underlying `web_kanidm_*_group` vars, because a shell `sed` cannot resolve Jinja.) ## Verification — each step, not by reasoning | Check | Result | | --- | --- | | `git apply --check` against a fresh v1.3.0 clone | clean | | `cargo build --release` on `rust:1-trixie` | succeeds | | field genuinely **parsed**, not ignored | `entryManagedBy: 12345` → `invalid type: integer 12345, expected a string` | | build through the **real Containerfile** (exercises `git apply`) | succeeds | | patched binary parses a realistic state with `entryManagedBy` | yes (fails only at the bogus URL) | | render the state template offline | valid JSON; `entryManagedBy` on **exactly** the 4 managed groups; **no key** on the other 12 | | `--syntax-check`, `ansible-lint` | clean at `production` | That wrong-type test is worth noting: it is precisely the test that would have caught the dead `service-accounts` block (#267). A **known** field validates its type; an **unknown** one is silently dropped. `git apply` rather than `patch`, deliberately — a context drift on a future ref bump fails the build loudly instead of half-applying. ## What stays The health gate and `signup-drill.sh` remain. They become belt-and-braces rather than the only defence — and **the gate is what will catch it if this patch is ever dropped** on a ref bump. The manual command is kept in the runbook as documented break-glass. ## Apply note This bumps the image tag, so the next apply **rebuilds the image on the host** (native Rust compile, ~3 min on prod based on the last one). Being offered upstream separately; there is no existing issue or PR for the feature.

supernaut lade till 3 incheckningar 2026-07-30 20:54:38 +00:00

fix(kanidm): detect real changes, and make a failed build retry instead of hiding ... 93d4c0a752

Three fixes in the kanidm-provision path, all found by researching upstream after
#265 landed.

1. changed_when: false was needlessly blind — a change signal DOES exist

#265 set `changed_when: false` on the reasoning that the tool emits no change
signal. That conclusion came from grepping the compiled binary for Created /
Updated / Modified — PAST TENSE. The tool uses GERUNDS. The grep could not match,
so a usable signal was missed.

Upstream reality: `update_entity_attrs()` wraps every mutation in
`if current_values != values`, so when values match, no request is sent and nothing
is printed. Only mutations emit a log_event, and only four verbs exist: Creating,
Updating, Deleting, Appending. None of the six unconditional phase headers contains
any of them, so a verb match is sound on v1.3.0 with no upstream change.

  changed_when: stdout is search('(Creating|Updating|Deleting)')

`Appending` is deliberately excluded. In append mode the comparison is against the
DECLARED members only, and every group here sets overwriteMembers:false — so any
group whose real membership is a superset of what we declare (idm_people_admins,
where we declare the service account but Kanidm also holds idm_admin) compares
unequal forever and logs Appending on every run. Same for the "Tracking provisioned
entities" step, which appends to ext_idm_provisioned_entities. Matching it would
pin changed:true permanently — recreating the bug #265 fixed.

Unanchored `search` on purpose: log_event colourises unconditionally (no
supports_color check), so ANSI escapes are present even when stdout is a pipe, and
the {:>12} padding is computed over the escape-laden string.

2. A failed build was invisible on retry

The Containerfile is staged BEFORE the build and `is changed` was the rebuild
trigger, so it is true only on the run that stages it. If that build then failed —
a Rust compile on a 2 vCPU / 4 GB host, and the Containerfile itself warns a link
step can be OOM-killed — the next apply found the staged file unchanged AND the old
image still tagged, silently skipped the build, and provisioned against the stale
image. A failure did not self-heal and became invisible on retry: the same shape as
the merged-but-unapplied Caddyfile in #250.

Now the trigger compares the staged Containerfile against a `.built-Containerfile`
marker written only AFTER the build task succeeds. A crashed build leaves the
mismatch, so the next run retries.

The marker is written on `is not skipped`, not `is changed`: podman_image reports ok
(not changed) when the tag exists and needs no rebuild, which is the steady state —
gating on `is changed` would never write the marker and leave the gate open forever.

3. The whole provisioning block was invisible to --check

`_kanidm_provision_ready` depends on a `uri` reachability probe, and `uri` does not
run under --check. So the build context, Containerfile staging, image build, state
render and provisioning run were ALL skipped in every dry-run. That is how #265's
base-image change reached production without appearing in a single --check: the
dry-run showed changed=1 (an unrelated artefact) while the real apply rebuilt an
image. Same trap as #257.

The probe and the image-exists check are read-only, so both now run under --check.
podman_image declares supports_check_mode, so the build reports what it would do
rather than compiling, and the provisioning `command` is still skipped — a dry-run
never mutates Kanidm.

Verified: `--check --tags kanidm` now shows the probe, image check, both stats and
the build as ok (previously all skipped), and the marker task as changed.

Also corrected the CARGO_BUILD_JOBS hint: `podman build -e` does not exist, so the
previous comment was not copy-pasteable. It needs ARG+ENV in the Containerfile or
--build-arg.

fix(kanidm): remove the silently-discarded service-accounts block; correct the claim ... 7a3ae4d0bd

`kanidm-state.json.j2` declared a `"service-accounts"` block, and the role defaults
stated "The account + its group memberships are declared in kanidm-state.json.j2 so
kanidm-provision manages them."

Half of that was false. The block was DEAD. Verified against upstream v1.3.0
`src/state.rs`:

  #[derive(Debug, Deserialize)]
  #[serde(rename_all = "camelCase")]
  pub struct State { pub groups: …, pub persons: …, pub systems: … }

Three fields, and no `#[serde(deny_unknown_fields)]` — so serde parsed
`"service-accounts"`, found no matching field, and discarded it without a warning.
Service-account support is upstream PR #29, unreviewed for 12 months.

The template's own comment already flagged this as unresolved: "⚠️ VERIFY the exact
key name "service-accounts" and the object shape against kanidm-provision's schema".
That verification was never done, and the answer was "it does not exist".

What is actually true:

- The service account's GROUP MEMBERSHIPS are managed — _sa_group_members merges it
  into the `groups` block, and the rendered state declares gitborg-web-provision as a
  member of idm_people_on_boarding, idm_people_admins and idm_gitborg_ent_managers.
- The ACCOUNT OBJECT is not. It is a manual bootstrap, created by hand at the same
  time as its API token.

This is the same shape of latent gap as the entry-managed-by delegation that broke
sign-up for 13 days: something assumed declarative that never was. Worth removing
rather than leaving in place looking managed.

Also documents two things in the runbook:

- A rebuild-ordering hazard: because `groups` names the account as a member, a fresh
  host must have the service account created BEFORE this role provisions, or those
  member writes reference a nonexistent entity. Full rebuild sequence added.
- `"present": false` DELETES regardless of `--no-auto-remove` — that flag only gates
  removal of orphans (entities dropped from the state file). Nothing here uses
  present:false today; the note exists so nobody reaches for it expecting protection.

Verified by rendering the template offline with representative vars: the output is
valid JSON, its top-level keys are exactly groups/persons/systems (matching upstream's
struct, so nothing is silently dropped any more), `service-accounts` is gone, and the
service account is still declared as a member of the three privilege groups.

feat(kanidm): make the entry-manager delegation declarative via a carried patch ... 30ca733b2b

Retires the manual `kanidm group set-entry-manager` step whose omission broke
sign-up for 13 days (2026-07-17..30). ADR 0029 moved the tier group and ADR 0036
added the trial group; the manual delegation followed neither, and Kanidm reports a
denied member-write as 404, so the symptom was a bare "group add -> 404".

The premise that blocked this was wrong. "kanidm-provision has no ACP support" is
true but irrelevant: gitborg does not need access-control PROFILES. Kanidm's
`kanidm group set-entry-manager` is one ordinary attribute write —

  PUT /v1/group/<group>/_attr/entry_managed_by  ["<manager>"]

— which is exactly the request `update_entity_attrs()` already issues, as the
idm_admin it already authenticates as. The blocker was a missing optional field on
`struct Group`, not an architecture.

So: a carried patch on the pinned upstream ref, in three parts.

- state.rs: `entry_managed_by: Option<String>` on Group (camelCase =>
  `entryManagedBy`). Omitted/null means LEAVE ALONE.
- client.rs: extend the SPN-stripping that already exists for `member` to
  `entry_managed_by`. Kanidm returns it as "name@domain", so without this the
  comparison never matches what we wrote and the attribute is rewritten every run.
  kaniop independently normalises the same attribute the same way.
- main.rs: write it in the "Syncing group members" phase, NOT in sync_groups(),
  because state.groups is an unordered HashMap — by that phase every group exists, so
  the manager can safely be another provisioned group.

The `if let Some(...)` guard is load-bearing. update_entity_attrs() treats an empty
Vec with append=false as the `values.is_empty()` case and issues a DELETE, so passing
an empty list for an undeclared field would STRIP an entry manager set outside this
tool. Omission must mean "leave alone", never "remove".

Also decouples two things that were conflated: kanidm_provision_image_tag doubled as
the upstream git ref, so a patched image could not be labelled distinctly without
breaking the clone. Now kanidm_provision_upstream_ref (what to clone) and
kanidm_provision_image_tag ("v1.3.0-gitborg1", bumped when the patch changes).

And collapses THREE copies of the delegated-group list into one
(kanidm_portal_managed_groups in group_vars/all), consumed by the state template and
by site.yml's health gate. Three copies that had to agree is the root shape of the
outage; the drill reads the same underlying web_kanidm_*_group vars because a shell
sed cannot resolve Jinja.

Verified, each step rather than by reasoning:

- patch applies cleanly to a fresh v1.3.0 clone (`git apply --check`)
- `cargo build --release` succeeds on rust:1-trixie
- the field is genuinely PARSED, not silently ignored: entryManagedBy: 12345 fails
  with "invalid type: integer `12345`, expected a string". (That same test would have
  caught the dead service-accounts block — a known field validates its type, an
  unknown one is dropped.)
- building through the real Containerfile succeeds, exercising `git apply` in the flow
- the patched binary parses a realistic state containing entryManagedBy
- rendering the state template offline: valid JSON, entryManagedBy on exactly the four
  managed groups, and NO key on the other 12 — so their existing entry manager is
  untouched
- --syntax-check and ansible-lint clean at the production profile

`git apply` (not `patch`) on purpose: a context drift on a future ref bump fails the
build loudly instead of half-applying.

The health gate and signup-drill stay. They become belt-and-braces rather than the
only defence — and the gate is what will catch it if this patch is ever dropped.

Offering upstream separately; there is no existing issue or PR for it.

supernaut lade till etiketterna

area/identity

type

feature

2026-07-30 20:54:38 +00:00

supernaut tvångsskickade feat/kanidm-declarative-entry-manager från 30ca733b2b till 825b513443 2026-07-30 20:59:24 +00:00 Jämför

supernaut tvångsskickade feat/kanidm-declarative-entry-manager från 825b513443 till 36c1207ae6 2026-07-30 21:06:04 +00:00 Jämför

supernaut sammanfogade incheckning d732602ad1 till main 2026-07-30 21:15:02 +00:00

supernaut tog bort grenen feat/kanidm-declarative-entry-manager 2026-07-30 21:15:02 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning 2026-07-30 21:15:03 +00:00

feat(kanidm): make the entry-manager delegation declarative via a carried patch (#268)

supernaut refererade till denna ändringsförfrågan 2026-07-30 21:35:30 +00:00

fix(docs): restore the runbook — my regex in #268 deleted 2203 lines #269

supernaut refererade denna ändringsförfrågan från en incheckning 2026-07-30 21:35:31 +00:00

fix(docs): restore the runbook — my regex in #268 deleted 2203 lines

supernaut refererade denna ändringsförfrågan från en incheckning 2026-07-30 21:41:00 +00:00

fix(docs): restore the runbook — my regex in #268 deleted 2203 lines (#269)

supernaut refererade till denna ändringsförfrågan 2026-07-30 22:41:37 +00:00

Incident 2026-07-17..30: sign-up broken for 13 days — Kanidm entry-manager delegation never followed ADR 0029/0036 #271

supernaut refererade denna ändringsförfrågan från en incheckning 2026-08-03 09:41:35 +00:00

feat(kanidm): make the entry-manager delegation declarative via a carried patch (#268)

supernaut refererade denna ändringsförfrågan från en incheckning 2026-08-03 09:41:35 +00:00

fix(docs): restore the runbook — my regex in #268 deleted 2203 lines (#269)

Logga in för att delta i denna konversation.

Granskare

Inga granskare

Etiketter

Rensa etiketter

  

area/backups

Backup & restore

  

area/ci

Actions / runners

  

area/control-panel

Logged-in self-service area

  

area/identity

Kanidm / OIDC / entitlements

  

area/infra

Infrastructure (OpenTofu/Ansible/host)

  

area/observability

Monitoring / logs / analytics

  

area/payments

Billing / payment providers

  

area/security

Security & compliance (incl. ISO 27001)

  

area/storage

Volumes / data / LFS

  

area/web

Public web portal (gitborg-web)

  

blocked

Evaluated and specified, but waiting on an external dependency

  

needs-info

Waiting on the reporter for more information

  

needs-triage

Not yet evaluated

  

ready-for-implementation

Fully specified, ready to be picked up

  

type

bug

Something is broken

  

type

chore

Maintenance / housekeeping

  

type

docs

Documentation

  

type

epic

High-level roadmap item

  

type

feature

New capability

  

type

task

Concrete unit of work

  

wontfix

Will not be actioned

Inga etiketter

area/backups

area/ci

area/control-panel

area/identity

area/infra

area/observability

area/payments

area/security

area/storage

area/web

blocked

needs-info

needs-triage

ready-for-implementation

type

bug

type

chore

type

docs

type

epic

type

feature

type

task

wontfix

Milstolpe

Rensa milstolpe

Inga objekt

Ingen milstolpe

Projekt

Rensa projekt

Inga objekt

Inget projekt

Tilldelade

Rensa tilldelade

Inga tilldelade

1 deltagare

Notiser

Prenumerera

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra!268

Ingen beskrivning angiven.