kanidm: 1.10.x leaves support around 2026-09-01, upgrade to 1.11.1 #424
Etiketter
Inga etiketter
area/backups
area/ci
area/control-panel
area/identity
area/infra
area/observability
area/payments
area/security
area/storage
area/web
blocked
needs-info
needs-triage
ready-for-implementation
type
bug
type
chore
type
docs
type
epic
type
feature
type
task
wontfix
Ingen milstolpe
Inget projekt
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Inget förfallodatum satt.
Beroenden
Inga beroenden satta
Referens
bitborg/bitborg-infra#424
Läser in…
Hänvisa till i nytt ärende
Ingen beskrivning angiven.
Ta bort grenen "%!s()"
Borttagning av en gren är permanent. Även om den borttagna grenen kan fortsätta existera en kort tid innan den faktiskt tas bort, kan det INTE ångras i de flesta fall. Vill du fortsätta?
Deadline
Production runs Kanidm 1.10.4 (
ansible/roles/kanidm/defaults/main.yml:12). Current stable is1.11.1, released 2026-08-14.
Kanidm supports each stable release for 4 months from its release date, on a quarterly cadence
(1 Feb, 1 May, 1 Aug, 1 Nov). Verified at
https://kanidm.github.io/kanidm/stable/support.html: "Stable releases will be supported for 4
months after their release date. This allows a 1 month support overlap between N and N+1 versions."
1.10.0 shipped 2026-05-01. Support ends around 2026-09-01. That is roughly two weeks out.
Why this cannot be deferred
Three constraints compound, and all three are enforced in code, not just documented.
stable release." The server raises
MG0008SkipUpgradeAttemptedrather than migrating.MG0010DowngradeNotAllowed. The project states this is an architecturalconclusion, not an oversight.
mismatched restore fails with
DB0001MismatchedRestoreVersion. A 1.10 backup restores onto any1.10.x and never onto 1.11.
1.12.0 is due 2026-11-01. If we are still on 1.10.x then, the direct path is gone. It becomes
1.10 to 1.11 to 1.12, with a stop-the-world step at each hop on a single-node deployment.
Plan
Take the latest patch of the current series first. 1.10.3's release notes fix "an incorrect
internal query that can prevent upgrades from 1.9 to 1.10", which is the pattern.
RELEASE_NOTES.md: it carries onlyX.Y.0entries and has drifted from the release bodies.kanidmd domain upgrade-checkand clear anything it flags.rollback is "start the previous container", with restore-from-backup as the fallback.
next tick projecting unchanged.
Known 1.11.0 breaking change
Passwords are capped at 128 UTF-8 characters / 512 bytes. Unlikely to matter under SSO-only, but
worth confirming no service account exceeds it.
Check alongside
override.cssselectors and the concealment health gate against 1.11 markup.kanidm-provisioncompatibility. Its patch set targets Kanidm 1.7 and 1.8, and it has had nocommits since 2025-11-22. See the separate issue for replacing it.
Standing consequence
This is not a one-off. The 4-month window plus the no-skip rule means four forced upgrades per
year, each a stop-the-world event. That cost is inherent to running Kanidm and should be planned
for rather than rediscovered each quarter.
Correction: one hop, not two. PR #425 stages it.
The plan above says 1.10.4 to 1.10.5 to 1.11.1. Having read the actual release bodies, the
intermediate hop is unnecessary.
The upgrade policy supports one minor behind current stable, so 1.10 to 1.11 is a direct
supported step. The reason to take the latest patch of the current series first is the 1.10.3
precedent, where a patch fixed "an incorrect internal query that can prevent upgrades from 1.9 to
1.10". Nothing of that kind exists in 1.10.5.
Read from the GitHub release bodies, not
RELEASE_NOTES.md:1.10.5 therefore has nothing 1.11.1 lacks, and the extra hop is one more database migration and one
more restart for no gain.
What the upgrade actually delivers
AccessTokenResponsesThe OAuth2 entries matter because both the git host and the portal authenticate through that
surface. They are fixes, so the expected effect is neutral-to-better, but exercise both logins
after the apply.
Revised checklist
kanidmd domain upgrade-check, clear anything it flags.characters / 512 bytes).
override.cssselectors and the concealment health gate still hold against 1.11 markup.The 1.10.4 to 1.10.5 step is struck.
Rollback, corrected
The plan above says to keep the current image tag recoverable. Worth stating why, because it is
counter-intuitive: the rollback is to start the previous tag, not to restore the backup. A 1.10
backup cannot be loaded by 1.11 (
DB0001MismatchedRestoreVersion). The backup protects againstdata loss, not against a bad upgrade.
Now covered by #425
The role had no check that the running container is on the pinned image, which the restart handler
had flagged as an outstanding follow-up since 2026-08-04. #425 adds a pull before the handler flush
and a verify after start, matching forgejo, postgres, caddy and web.
Without it, a tag bump that silently no-ops would leave the operator believing a migration ran when
it did not, and planning the next hop from a version the server is not on.
Risk to watch
kanidm-provisionis pinned atv1.3.0, built from source with a carriedentryManagedBypatch.Upstream has had no commits since 2025-11-22, and its own patch set targets Kanidm 1.7 and 1.8.
Compatibility with a 1.11 server is unverified. If entitlement groups stop converging after the
upgrade, look there first.
Applied to production 2026-08-16. Kanidm is on 1.11.1.
One hop, as corrected above. Applied together with the merged backup-tooling bumps and some
pre-existing rename drift, via
./scripts/apply-reconcile.sh.Result
A second
--checkafterwards returnedchanged=0, which is the only thing that proves prodmatches main.
Verified
docker.io/kanidm/server:1.11.1(was 1.10.4)kanidmunitokgit.bitborg.se/api/healthzauth.bitborg.se/statuswww.bitborg.se/bitborg_reconciler_last_run_statusPre-flight: backup 16.6 h old and restore-verified 15.6 h old, both taken at 1.10.x so they were
restore-compatible with the version being replaced.
The flagged risk did not materialise
kanidm-provisionis pinned atv1.3.0with a patch set targeting Kanidm 1.7/1.8, and upstream hashad no commits since 2025-11-22. It is invisible to
--checkbecause it is acommandtask, so itwas the one thing the dry run could not clear. It ran and returned
okagainst the 1.11 server.That risk is retired for 1.11. It is not retired for 1.12, and the tool is still dormant upstream.
Two things still open
1. The ADR 0038 concealment gate was overridden for this run.
The gate correctly blocked the apply:
Kanidm: running 1.11.1, selectors verified against 1.10.4.There is a chicken-and-egg, since 1.11's markup cannot be inspected until 1.11 is running.
It was overridden at runtime with
-e health_check_concealment=false, not by committing adisabled gate or by pre-bumping the verified tag, so nothing in the repository asserts a
verification that has not happened.
The six surfaces now need checking by hand against the running 1.11.1, per the gate's own list:
/user/settings→ no "Full Name" field/user/settings/account→ no "Set as primary", no password section, no deleteauth.bitborg.se/ui/profile→ no username / display name inputsauth.bitborg.se/ui/login→ the "No account yet?" note is present, signed outwww.bitborg.sesigned out → "Create account" in the navbarauth.bitborg.se/ui/reset?token=<t>→ the passkey NAME input is visible (the over-matchdirection, which is what broke passkey enrolment on 2026-08-04)
Then bump
health_check_kanidm_concealment_verified_tagto1.11.1and re-apply with the gate on.Until that happens the estate is running unverified concealment selectors.
2. A local
.netrcbreaks Ansibleuritasks.bad follower token 'region' (~/.netrc, line 4)makes everyuritask delegated to localhost failwith
Status code was -1, which killed the first dry run atroles/forgejo/tasks/system-webhook.yml. Worked around withNETRC=pointed at an empty file. Itis an operator-workstation problem, not a repository one, but it will hit anyone whose
.netrccarries a non-netrc token.
Closing: the upgrade is complete and both flagged items are resolved
Kanidm has run 1.11.1 in production since 2026-08-16, and the two things the apply comment left
open are now closed. Recording the evidence here so the state is not carried in anyone's head.
1. The ADR 0038 concealment gate is back on
Closed by #436 (
e675d44, 2026-08-17). All six surfaces were checked by hand, in both directionsthe gate cares about, including the over-match direction that broke passkey enrolment on 2026-08-04.
health_check_kanidm_concealment_verified_tagis now1.11.1, matchingkanidm_image_tag, and afull
--checkwith the gate on and no override returned:The comment above said "until that happens the estate is running unverified concealment selectors".
That is no longer true, and this note supersedes it.
2. The
~/.netrcbreakage no longer affects AnsibleClosed by #434 (
4270f87, 2026-08-17): everyansible.builtin.uritask now carriesuse_netrc: false. It remains an operator-workstation problem for other tools, but it can no longerkill a play.
The revised checklist, honestly
Not ticking the list above item by item, because three of its entries were never recorded and
ticking them would assert a verification nobody can point at.
changed=7, matching the dry run exactly; second--checkreturnedchanged=0ok, and the running image reads1.11.1bitborg_reconciler_last_run_status 0override.css+ concealment gate hold on 1.11kanidmd domain upgrade-checkgit.bitborg.se/api/healthz,auth.bitborg.se/statusandwww.bitborg.se/all returned 200, which is not the same as an interactive sign-in. Note that no probe covers/user/loginat all, which is #384Closing on the substance: the deadline this issue existed for (1.10.x leaving support around
2026-09-01) is retired, the server is on a supported release, and the gate that guards the next one
is armed. The two genuinely unaudited items above are small and non-blocking; the sign-in coverage
gap is tracked separately as #384.
Carried forward, not closed here
kanidm-provisionreturnedokagainst the 1.11 server, so that risk is retired for 1.11only. It is still pinned at
v1.3.0with a patch set targeting Kanidm 1.7/1.8 and no upstreamcommits since 2025-11-22.
upgrade mandatory rather than optional. This recurs four times a year.
as #441.