web role: rotating a podman secret reports success but does not restart the container #274

Nytt ärende

Stängd

öppnade 2026-07-31 09:34:32 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-07-31 09:34:32 +00:00

Ägare

Rotating any of the web role's podman secrets reports changed and does not restart the
container, so the running service keeps using the old value while Ansible reports success.

Observed

During the Sweego API key rotation (#273) the apply produced:

TASK [web : Create the web Sweego API key podman secret] ***
changed: [gitborg-prod]

TASK [web : Enable and start bitborg-web (restart on unit change or image drift)] ***
ok: [gitborg-prod]

PLAY RECAP … changed=1 failed=0 — a clean, successful-looking apply. But podman injects secrets
into the container environment at container creation, so the process was still running with the
previous key. The rotation only took effect after an explicit
systemctl --user restart bitborg-web.

Cause

roles/web/tasks/main.yml:295 keys the restart on two conditions, neither of which a secret
rotation satisfies:

state: >-
  {{ 'restarted'
     if (web_unit is changed)
        or ((web_running_image.stdout | default('') | trim) != (web_image ~ ':' ~ web_image_tag))
     else 'started' }}

started is a no-op on a running service. The unit file is unchanged because the web secrets use
static names, so rotating the secret's content leaves the unit byte-identical.

Why forgejo is not affected

The forgejo role embeds a content hash in the secret name, so a value change renames the secret,
which rewrites the unit, which trips web_unit is changed:

-Secret=bitborg-forgejo-mailer-passwd-8252d222965a,type=env,target=FORGEJO__mailer__PASSWD
+Secret=bitborg-forgejo-mailer-passwd-9d8af8be4016,type=env,target=FORGEJO__mailer__PASSWD

That is the pattern the web role is missing.

Scope

All seven web secrets are affected, not just the mail key:

DATABASE_URL · Kanidm provision token · Sweego API key · sign-up preview token · Cap captcha
secret · OIDC client secret · session signing secret

The consequences differ in severity but share the failure mode — a rotation that appears to
succeed and silently does not apply. For a credential being rotated deliberately, "Ansible said
changed" is not evidence the old value is out of service. Any past web-secret rotation should be
re-verified rather than assumed effective.

Aggravating factor: check mode cannot see it

Create the web … podman secret reports skipping under --check, because podman commands do not
run in check mode. So a dry-run shows neither the rotation nor its absence — the pre-apply gate is
structurally blind here, which is worth a comment in the role even after the fix.

Options

Content-hash the secret names (the forgejo pattern). Rotation renames the secret → unit
changes → restart. Consistent with existing code, no new mechanism. Downside: orphaned old
secrets accumulate unless pruned.
Notify a handler from each secret task, and restart if any fired. More explicit and avoids
name churn, but adds a second restart trigger alongside the inline state: expression, so the
two must be kept coherent.
Include a digest of the secret set in the unit file as a comment or label — one hash covering
all seven, so any rotation changes the unit exactly once.

Option 1 is the most consistent with the codebase; option 3 is tidier if orphan pruning is a
concern.

Done when

Rotating any web podman secret causes the container to be recreated in the same apply, and a test
proves the running process observes the new value.

Rotating any of the web role's podman secrets reports `changed` and **does not restart the container**, so the running service keeps using the old value while Ansible reports success. ## Observed During the Sweego API key rotation (#273) the apply produced: ``` TASK [web : Create the web Sweego API key podman secret] *** changed: [gitborg-prod] TASK [web : Enable and start bitborg-web (restart on unit change or image drift)] *** ok: [gitborg-prod] ``` `PLAY RECAP … changed=1 failed=0` — a clean, successful-looking apply. But podman injects secrets into the container environment at **container creation**, so the process was still running with the previous key. The rotation only took effect after an explicit `systemctl --user restart bitborg-web`. ## Cause `roles/web/tasks/main.yml:295` keys the restart on two conditions, neither of which a secret rotation satisfies: ```yaml state: >- {{ 'restarted' if (web_unit is changed) or ((web_running_image.stdout | default('') | trim) != (web_image ~ ':' ~ web_image_tag)) else 'started' }} ``` `started` is a no-op on a running service. The unit file is unchanged because the web secrets use **static names**, so rotating the secret's *content* leaves the unit byte-identical. ## Why forgejo is not affected The forgejo role embeds a content hash in the secret name, so a value change renames the secret, which rewrites the unit, which trips `web_unit is changed`: ``` -Secret=bitborg-forgejo-mailer-passwd-8252d222965a,type=env,target=FORGEJO__mailer__PASSWD +Secret=bitborg-forgejo-mailer-passwd-9d8af8be4016,type=env,target=FORGEJO__mailer__PASSWD ``` That is the pattern the web role is missing. ## Scope All seven web secrets are affected, not just the mail key: `DATABASE_URL` · Kanidm provision token · Sweego API key · sign-up preview token · Cap captcha secret · OIDC client secret · session signing secret The consequences differ in severity but share the failure mode — **a rotation that appears to succeed and silently does not apply.** For a credential being rotated deliberately, "Ansible said changed" is not evidence the old value is out of service. Any past web-secret rotation should be re-verified rather than assumed effective. ## Aggravating factor: check mode cannot see it `Create the web … podman secret` reports `skipping` under `--check`, because podman commands do not run in check mode. So a dry-run shows neither the rotation nor its absence — the pre-apply gate is structurally blind here, which is worth a comment in the role even after the fix. ## Options 1. **Content-hash the secret names** (the forgejo pattern). Rotation renames the secret → unit changes → restart. Consistent with existing code, no new mechanism. Downside: orphaned old secrets accumulate unless pruned. 2. **Notify a handler** from each secret task, and restart if any fired. More explicit and avoids name churn, but adds a second restart trigger alongside the inline `state:` expression, so the two must be kept coherent. 3. **Include a digest of the secret set in the unit file** as a comment or label — one hash covering all seven, so any rotation changes the unit exactly once. Option 1 is the most consistent with the codebase; option 3 is tidier if orphan pruning is a concern. ## Done when Rotating any web podman secret causes the container to be recreated in the same apply, and a test proves the running process observes the new value.

supernaut lade till etiketterna

area/infra

ready-for-implementation

type

bug

2026-07-31 09:35:00 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-31 09:35:37 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-31 09:40:12 +00:00

fix(web): restart the container when a podman secret is rotated

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det