docs(runbook): correct the SECRET_KEY rotation warning and document the TOTP lockout #293

Sammanfogat

supernaut sammanfogade 1 incheckning från docs/292-secret-key-totp-lockout in i main

2026-07-31 21:52:31 +00:00

supernaut kommenterad

2026-07-31 21:46:34 +00:00

Ägare

Refs #292. Documentation only — no role, template or variable is touched.

What went wrong

The rotation procedure already carried the right warning about SECRET_KEY, and it was written
from a real incident (#73/#82, 2026-07-17). It named both casualties. Then it waved one of them
off:

2FA secrets are also lost (moot while SSO-only).

That parenthetical is wrong, and it is the whole failure. "SSO-only" describes how people sign in
now; it says nothing about what is still sitting in the database from before. An account created
before the Kanidm cutover can still hold a Forgejo-native TOTP enrolment, and Forgejo walks into it
on the OIDC callback.

So the Actions-secret half of the warning was actioned at the time and the 2FA half was reasoned
away — and on 2026-07-31 it came due as a 500 on the TOTP prompt, with the account locked out:

2fa.go:64:TwoFactorPost() [E] UserSignIn: chacha20poly1305: message authentication failed
router: completed POST /user/two_factor … 500 Internal Server Error

Worth being precise about the failure mode, because it is easy to misread as user error: the AEAD
tag simply does not verify. It is not a wrong code, not clock skew, and not recoverable by trying
again with a better authenticator. No code can ever succeed.

Why it stayed hidden for two weeks

While a passkey is registered, Forgejo takes the WebAuthn branch and never decrypts the TOTP
secret. The breakage was total from the moment of rotation and completely invisible. Removing a
passkey is what forces the fallback — so the trigger looks unrelated to the cause, fourteen days
and two rotations (#216 as well) downstream.

That gap between cause and symptom is the argument for the diagram rather than another paragraph:
the point an operator needs is that one config change fans out to several ciphertext columns with
unrelated-looking symptoms.

What changed

Step 3 of "Rotate secrets" is reframed as a destructive migration of every ciphertext
column, not a config change — the old key is gone, so each affected row must be re-created or
deleted, never repaired. The two casualties are split into their own bullets so neither can be
skimmed past, the false reassurance is gone, and the 2FA bullet now carries the enumeration query
to run as part of rotating.

A new troubleshooting entry in the OIDC/login list, keyed on the symptom an operator actually
has in front of them — a 500 on the TOTP prompt after a successful OIDC sign-in — rather than on
the cause, which is the thing they do not know yet. It states plainly that no code will work, gives
the query that finds the row and the delete that clears it, and notes the two things easy to get
wrong afterwards:

no Forgejo restart is needed — the row is read per request;
the stuck browser session is parked in the "2FA pending" state, so cookies for git.gitborg.se
must be cleared before signing in again, or the fix looks like it did not work.

It also records that a recovery code still works without DB access, because those are stored
hashed (scratch_salt/scratch_hash) rather than encrypted — so that path never touches the
broken decrypt. Useful when the lockout happens to someone without shell access.

Why deleting is the fix, not disabling

The enrolment is not repairable and it is not wanted: sign-in is SSO-only, so MFA belongs to
Kanidm, and a native enrolment can only ever duplicate it while carrying this hazard. Deleting the
row is therefore the correct end state rather than a workaround — which is why the runbook says so
outright instead of leaving the operator to wonder whether they are papering over something.

Verification

markdownlint-cli2 and Prettier both clean via the pre-commit hook (the first attempt failed on
MD040 for the unlabelled log fence — fixed, not bypassed).
Both new cross-references resolve: #rotate-secrets and
#troubleshooting-lessons-learned-the-hard-way.
The recovery commands are not invented for the runbook — they use this repo's established
rootless idiom (G="sudo -u gitborg env XDG_RUNTIME_DIR=/run/user/2000", POSTGRES_USER read
from the container) already used by the runner-diagnosis section, so they are consistent with
what an operator has muscle memory for.

Follow-ups

Tracked in #292, deliberately out of scope here: auditing two_factor for other accounts holding
a pre-rotation enrolment (each one is a latent lockout waiting for its owner to drop a passkey),
teaching the 5.18 access review to flag them so this surfaces in a report instead of a lockout,
and the open question of whether native 2FA enrolment should be prevented outright now that
sign-in is SSO-only.

Refs #292. Documentation only — no role, template or variable is touched. ## What went wrong The rotation procedure already carried the right warning about `SECRET_KEY`, and it was written from a real incident (#73/#82, 2026-07-17). It named both casualties. Then it waved one of them off: > 2FA secrets are also lost **(moot while SSO-only)**. That parenthetical is wrong, and it is the whole failure. "SSO-only" describes how people sign in _now_; it says nothing about what is still sitting in the database from before. An account created before the Kanidm cutover can still hold a Forgejo-native TOTP enrolment, and Forgejo walks into it on the OIDC callback. So the Actions-secret half of the warning was actioned at the time and the 2FA half was reasoned away — and on 2026-07-31 it came due as a **500 on the TOTP prompt**, with the account locked out: ```text 2fa.go:64:TwoFactorPost() [E] UserSignIn: chacha20poly1305: message authentication failed router: completed POST /user/two_factor … 500 Internal Server Error ``` Worth being precise about the failure mode, because it is easy to misread as user error: the AEAD tag simply does not verify. It is not a wrong code, not clock skew, and not recoverable by trying again with a better authenticator. No code can ever succeed. ## Why it stayed hidden for two weeks While a passkey is registered, Forgejo takes the WebAuthn branch and never decrypts the TOTP secret. The breakage was total from the moment of rotation and completely invisible. Removing a passkey is what forces the fallback — so the trigger looks unrelated to the cause, fourteen days and two rotations (#216 as well) downstream. That gap between cause and symptom is the argument for the diagram rather than another paragraph: the point an operator needs is that one config change fans out to several ciphertext columns with unrelated-looking symptoms. ## What changed **Step 3 of "Rotate secrets"** is reframed as a **destructive migration of every ciphertext column**, not a config change — the old key is gone, so each affected row must be re-created or deleted, never repaired. The two casualties are split into their own bullets so neither can be skimmed past, the false reassurance is gone, and the 2FA bullet now carries the enumeration query to run as part of rotating. **A new troubleshooting entry** in the OIDC/login list, keyed on the symptom an operator actually has in front of them — a 500 on the TOTP prompt after a *successful* OIDC sign-in — rather than on the cause, which is the thing they do not know yet. It states plainly that no code will work, gives the query that finds the row and the delete that clears it, and notes the two things easy to get wrong afterwards: - **no Forgejo restart is needed** — the row is read per request; - the stuck browser session is parked in the "2FA pending" state, so cookies for `git.gitborg.se` must be cleared before signing in again, or the fix looks like it did not work. It also records that a **recovery code still works** without DB access, because those are stored hashed (`scratch_salt`/`scratch_hash`) rather than encrypted — so that path never touches the broken decrypt. Useful when the lockout happens to someone without shell access. ## Why deleting is the fix, not disabling The enrolment is not repairable and it is not wanted: sign-in is SSO-only, so MFA belongs to Kanidm, and a native enrolment can only ever duplicate it while carrying this hazard. Deleting the row is therefore the correct end state rather than a workaround — which is why the runbook says so outright instead of leaving the operator to wonder whether they are papering over something. ## Verification - `markdownlint-cli2` and Prettier both clean via the pre-commit hook (the first attempt failed on MD040 for the unlabelled log fence — fixed, not bypassed). - Both new cross-references resolve: `#rotate-secrets` and `#troubleshooting-lessons-learned-the-hard-way`. - The recovery commands are not invented for the runbook — they use this repo's established rootless idiom (`G="sudo -u gitborg env XDG_RUNTIME_DIR=/run/user/2000"`, `POSTGRES_USER` read from the container) already used by the runner-diagnosis section, so they are consistent with what an operator has muscle memory for. ## Follow-ups Tracked in #292, deliberately out of scope here: auditing `two_factor` for other accounts holding a pre-rotation enrolment (each one is a latent lockout waiting for its owner to drop a passkey), teaching the 5.18 access review to flag them so this surfaces in a report instead of a lockout, and the open question of whether native 2FA enrolment should be prevented outright now that sign-in is SSO-only.

supernaut lade till 1 incheckning

2026-07-31 21:46:35 +00:00

docs(runbook): correct the SECRET_KEY rotation warning and document the TOTP lockout

ci / ci (pull_request) Successful in 17s

Detaljer

454c236fad

Refs #292.

The rotation warning told the truth about Actions secrets and then waved off
the other half — "2FA secrets are also lost (moot while SSO-only)". That
parenthetical is wrong, and it is what turned a documented consequence into a
live lockout: accounts created before the Kanidm cutover still carry a native
TOTP enrolment, so the loss is not moot for them.

Rewrites step 3 as a destructive migration of every ciphertext column, with a
diagram of the blast radius, and splits the two casualties into their own
bullets so neither can be skimmed past. Adds a symptom-first troubleshooting
entry keyed on what an operator actually sees — a 500 on the TOTP prompt after
a successful OIDC sign-in — with the query that finds the row and the delete
that clears it.

supernaut lade till etiketterna

2026-07-31 21:46:40 +00:00

supernaut sammanfogade incheckning a1de1ec672 till main

2026-07-31 21:52:31 +00:00

supernaut tog bort grenen docs/292-secret-key-totp-lockout

2026-07-31 21:52:31 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-31 21:52:32 +00:00