registry-mirror: the Kanidm pin drifted to 1.10.4 and nothing can notice #441

Nytt ärende

Stängd

öppnade 2026-08-18 19:58:22 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-18 19:58:22 +00:00

Ägare

The drift

registry_mirror_images still pins Kanidm at 1.10.4. The role has run 1.11.1 since the
upgrade on 2026-08-16 (#424).

# ansible/roles/registry-mirror/defaults/main.yml:45
- src: "docker.io/kanidm/server:1.10.4"  # keep in lockstep with roles/kanidm/defaults (kanidm_image[_tag])
  dest: "bitborg/kanidm:1.10.4"

# ansible/roles/kanidm/defaults/main.yml:27
kanidm_image_tag: "1.11.1"

The comment says "keep in lockstep". It is not in lockstep, and nothing noticed for two days.

Why nothing noticed

Three properties compound, and each is individually reasonable.

1. Nothing consumes the mirrored image. bitborg/kanidm appears exactly once in the repository,
as its own dest. The Kanidm container pulls {{ kanidm_image }}:{{ kanidm_image_tag }} from
docker.io directly, so no consumer exists that could fail when the mirror entry is wrong. Contrast
renovate_image, which is {{ bitborg_registry_host }}/bitborg/renovate:43 and therefore points AT
the mirror: a mismatch there fails loudly with a missing image. The Kanidm entry is unverifiable by
construction.

2. The smoke's mirror lookup degrades to silence. load_mirror_map() keys the map on the exact
upstream ref, and pull_image() does mirror_map.get(image). A tag mismatch is not a miss, it is a
None, which means "this image has no mirror candidate" and is documented as never an error. So the
smoke silently falls back to upstream. That is correct behaviour for a genuinely unmirrored image
and wrong only here, where the image IS mirrored under a stale tag.

3. Renovate cannot catch it. kanidm_image_tag is renovate-exempt by deliberate decision, as
the server refuses minor skips and downgrades. So the one bot that would otherwise bump both
literals together will never look at either.

Observed

From today's CI run on main:

PASS  roles/forgejo/templates/forgejo.container.j2   git.bitborg.se/bitborg/forgejo:16.0.1-rootless [mirror]
PASS  roles/postgres/templates/postgres.container.j2 git.bitborg.se/bitborg/postgres:17.10-trixie   [mirror]
PASS  roles/kanidm/templates/kanidm.container.j2     docker.io/kanidm/server:1.11.1                 [upstream]

Forgejo and postgres resolve to the instance. Kanidm reaches docker.io. That is the sovereignty and
CI-resilience goal of #137 quietly not being met for one image, plus a stale 1.10.4 image occupying
registry space that nothing will ever pull.

Options

A. A static check (recommended, cheap). Assert that every literal src in
registry_mirror_images matches the role default it claims lockstep with. Same shape as
check-renovate-annotations.py and check-metric-names.py: a pure text-adjacency property, no
production access, runs in CI on the ansible gate. It also generalises, so the next literal added
to the list is covered without anyone remembering to add a check.

B. Move kanidm_image / kanidm_image_tag into group_vars/all/vars.yml and template the
mirror entry the way forgejo, postgres and caddy already are, removing the duplicated literal
entirely. Structurally the better answer, but it has real blast radius:

The # renovate-exempt: comment must move with the variable and stay directly adjacent, or
check-renovate-annotations.py reads differently than intended.
site.yml turns the ADR 0038 concealment gate off on the monitoring host because
kanidm_image_tag is a role default and therefore undefined in that play. Promoting it to a
group_var changes that reasoning, and the comment there would become wrong.

A is the fix for the bug. B is a separate refactor and should not ride along on it.

Done when

Changing a role's image tag without updating its mirror entry fails CI, and that failure has been
observed rather than assumed.

Notes

Fixing the pin alone is not sufficient. Without a check this recurs at every Kanidm upgrade, and
there are four forced upgrades a year (#424).
The mirror runs weekly (Sun *-*-* 04:30) and once on deploy, so the corrected entry populates on
the next apply.

## The drift `registry_mirror_images` still pins Kanidm at **1.10.4**. The role has run **1.11.1** since the upgrade on 2026-08-16 (#424). ```yaml # ansible/roles/registry-mirror/defaults/main.yml:45 - src: "docker.io/kanidm/server:1.10.4" # keep in lockstep with roles/kanidm/defaults (kanidm_image[_tag]) dest: "bitborg/kanidm:1.10.4" # ansible/roles/kanidm/defaults/main.yml:27 kanidm_image_tag: "1.11.1" ``` The comment says "keep in lockstep". It is not in lockstep, and nothing noticed for two days. ## Why nothing noticed Three properties compound, and each is individually reasonable. **1. Nothing consumes the mirrored image.** `bitborg/kanidm` appears exactly once in the repository, as its own `dest`. The Kanidm container pulls `{{ kanidm_image }}:{{ kanidm_image_tag }}` from docker.io directly, so no consumer exists that could fail when the mirror entry is wrong. Contrast `renovate_image`, which is `{{ bitborg_registry_host }}/bitborg/renovate:43` and therefore points AT the mirror: a mismatch there fails loudly with a missing image. The Kanidm entry is unverifiable by construction. **2. The smoke's mirror lookup degrades to silence.** `load_mirror_map()` keys the map on the exact upstream ref, and `pull_image()` does `mirror_map.get(image)`. A tag mismatch is not a miss, it is a `None`, which means "this image has no mirror candidate" and is documented as never an error. So the smoke silently falls back to upstream. That is correct behaviour for a genuinely unmirrored image and wrong only here, where the image IS mirrored under a stale tag. **3. Renovate cannot catch it.** `kanidm_image_tag` is `renovate-exempt` by deliberate decision, as the server refuses minor skips and downgrades. So the one bot that would otherwise bump both literals together will never look at either. ## Observed From today's CI run on `main`: ```text PASS roles/forgejo/templates/forgejo.container.j2 git.bitborg.se/bitborg/forgejo:16.0.1-rootless [mirror] PASS roles/postgres/templates/postgres.container.j2 git.bitborg.se/bitborg/postgres:17.10-trixie [mirror] PASS roles/kanidm/templates/kanidm.container.j2 docker.io/kanidm/server:1.11.1 [upstream] ``` Forgejo and postgres resolve to the instance. Kanidm reaches docker.io. That is the sovereignty and CI-resilience goal of #137 quietly not being met for one image, plus a stale 1.10.4 image occupying registry space that nothing will ever pull. ## Options **A. A static check (recommended, cheap).** Assert that every literal `src` in `registry_mirror_images` matches the role default it claims lockstep with. Same shape as `check-renovate-annotations.py` and `check-metric-names.py`: a pure text-adjacency property, no production access, runs in CI on the `ansible` gate. It also generalises, so the next literal added to the list is covered without anyone remembering to add a check. **B. Move `kanidm_image` / `kanidm_image_tag` into `group_vars/all/vars.yml`** and template the mirror entry the way forgejo, postgres and caddy already are, removing the duplicated literal entirely. Structurally the better answer, but it has real blast radius: - The `# renovate-exempt:` comment must move with the variable and stay directly adjacent, or `check-renovate-annotations.py` reads differently than intended. - `site.yml` turns the ADR 0038 concealment gate off on the monitoring host **because** `kanidm_image_tag` is a role default and therefore undefined in that play. Promoting it to a group_var changes that reasoning, and the comment there would become wrong. A is the fix for the bug. B is a separate refactor and should not ride along on it. ## Done when Changing a role's image tag without updating its mirror entry fails CI, and that failure has been observed rather than assumed. ## Notes - Fixing the pin alone is not sufficient. Without a check this recurs at every Kanidm upgrade, and there are four forced upgrades a year (#424). - The mirror runs weekly (`Sun *-*-* 04:30`) and once on deploy, so the corrected entry populates on the next apply.

supernaut lade till etiketterna

area/ci

area/infra

ready-for-implementation

type

bug

2026-08-18 19:58:28 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-18 19:58:41 +00:00

supernaut refererade till detta ärende

2026-08-18 20:00:13 +00:00

kanidm: 1.10.x leaves support around 2026-09-01, upgrade to 1.11.1 #424