smoke-containers: pull base images from the in-instance registry mirror (CI resilience + sovereignty) #137

Nytt ärende

Stängd

öppnade 2026-07-19 09:23:26 +00:00 av supernaut · 3 kommentarer

supernaut kommenterad 2026-07-19 09:23:26 +00:00

Ägare

Kopiera länk

Severity: MEDIUM (CI reliability + sovereignty). Found when PR #136's smoke-containers job failed on a transient codeberg.org 504 while pulling forgejo:16.0.0.

Problem

scripts/smoke-containers.py pulls base images live from external registries (codeberg.org, docker.io) on every CI run. So:

• an upstream registry blip fails CI (false negative — nothing to do with the PR), and
• it's a sovereignty wrinkle (principle 1): CI reaches outside the instance for images every run.

We already run a registry-mirror role (ADR 0023) that skopeo copys pinned images into the Forgejo registry (git.gitborg.se/bitborg/<name>:<tag>), but it currently mirrors only Renovate, and smoke-containers.py explicitly skips git.gitborg.se/… images as "not pullable in CI".

Fix

1. Mirror the smoke base images — add forgejo / postgres / caddy / kanidm to registry_mirror_images (pinned, in lockstep with the consuming roles' tags).
2. smoke pulls mirror-first with fallback + retry — for each public image, try the mirror ref first, fall back to the upstream ref, each with a small retry/backoff. Fallback keeps it non-breaking while the mirror is being populated; retry alone already fixes the transient-504 class.
3. Make the mirror CI-pullable (the gating dependency) — the smoke runner must be able to podman pull git.gitborg.se/bitborg/<name>:<tag>. Either mark those org packages public (base images are public upstream anyway → anonymous pull) or add a scoped podman login in the CI workflow. Until this lands, mirror-first simply falls back to upstream (so step 2's retry is what fixes the flake now).

Rollout

• Role change + site.yml apply → mirror timer populates the 4 packages.
• Set package visibility (or CI login).
• Then smoke prefers the mirror; external outages stop breaking CI.

Refs: PR #136 (the flake), registry-mirror role, ADR 0023, smoke #104.

**Severity: MEDIUM (CI reliability + sovereignty).** Found when PR #136's `smoke-containers` job failed on a transient `codeberg.org` **504** while pulling `forgejo:16.0.0`. ## Problem `scripts/smoke-containers.py` pulls base images **live from external registries** (codeberg.org, docker.io) on every CI run. So: - an upstream registry blip fails CI (false negative — nothing to do with the PR), and - it's a sovereignty wrinkle (principle 1): CI reaches outside the instance for images every run. We already run a **`registry-mirror`** role (ADR 0023) that `skopeo copy`s pinned images into the Forgejo registry (`git.gitborg.se/bitborg/<name>:<tag>`), but it currently mirrors **only Renovate**, and `smoke-containers.py` explicitly **skips** `git.gitborg.se/…` images as "not pullable in CI". ## Fix 1. **Mirror the smoke base images** — add forgejo / postgres / caddy / kanidm to `registry_mirror_images` (pinned, in lockstep with the consuming roles' tags). 2. **smoke pulls mirror-first with fallback + retry** — for each public image, try the mirror ref first, fall back to the upstream ref, each with a small retry/backoff. Fallback keeps it non-breaking while the mirror is being populated; retry alone already fixes the transient-504 class. 3. **Make the mirror CI-pullable** (the gating dependency) — the smoke runner must be able to `podman pull git.gitborg.se/bitborg/<name>:<tag>`. Either mark those org packages **public** (base images are public upstream anyway → anonymous pull) or add a scoped `podman login` in the CI workflow. Until this lands, mirror-first simply falls back to upstream (so step 2's retry is what fixes the flake now). ## Rollout - Role change + `site.yml` apply → mirror timer populates the 4 packages. - Set package visibility (or CI login). - Then smoke prefers the mirror; external outages stop breaking CI. Refs: PR #136 (the flake), `registry-mirror` role, ADR 0023, smoke #104.

supernaut lade till etiketterna

area/ci

area/infra

ready-for-implementation

type

task

2026-07-19 09:23:41 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure 2026-07-19 09:23:43 +00:00

supernaut refererade till detta ärende från en incheckning 2026-07-19 09:27:01 +00:00

fix(ci): smoke-containers pulls base images mirror-first with fallback + retry (#137)

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det 2026-07-19 09:27:20 +00:00

fix(ci): smoke-containers pulls base images mirror-first with fallback + retry (#137) #138

supernaut stängde ärendet 2026-07-19 09:31:39 +00:00

supernaut refererade till detta ärende från en incheckning 2026-07-19 09:31:41 +00:00

fix(ci): smoke-containers pulls base images mirror-first with fallback + retry (#137) (#138)

supernaut refererade till detta ärende från en incheckning 2026-07-19 09:37:18 +00:00

ci: best-effort registry login so smoke pulls from the mirror (#137)

supernaut refererade till detta ärende 2026-07-19 09:37:37 +00:00

ci: best-effort registry login so smoke pulls from the mirror (#137) #139

supernaut kommenterad 2026-07-19 09:38:24 +00:00

Upphovsperson

Ägare

Kopiera länk

Implementation status

• Mirror the base images — PR #138 (merged): registry-mirror now includes forgejo/postgres/caddy/kanidm.
• smoke pulls mirror-first + fallback + retry — PR #138 (merged).
• CI registry login — PR #139 (open): best-effort podman login git.gitborg.se before smoke, non-fatal.

Operator steps to activate (needs prod / out-of-band)

1. Apply so the mirror timer populates the 4 packages (also picks up the merged #136 controller fix):

   cd ansible && ansible-playbook site.yml --check --diff   # preview
   ansible-playbook site.yml
   

   Verified locally: pnpm ansible:check passes; registry_mirror_images has 5 entries on main.
2. Mint a read-only token + set the Actions secret (least-privilege — read:package only, distinct from the mirror's write token):

   # create a read:package PAT on gitborg-ci, then:
   fj actions secrets set REGISTRY_READ_TOKEN --repo gitborg/gitborg-infra
   

3. Merge #139.

Anonymous pull is off (git.gitborg.se/v2/ → 401) and the gitborg org must stay private, hence the login rather than public packages. Until 1–3 are done, smoke keeps pulling upstream (already resilient via the #138 retry).

## Implementation status - [x] **Mirror the base images** — PR #138 (merged): registry-mirror now includes forgejo/postgres/caddy/kanidm. - [x] **smoke pulls mirror-first + fallback + retry** — PR #138 (merged). - [x] **CI registry login** — PR #139 (open): best-effort `podman login git.gitborg.se` before smoke, non-fatal. ## Operator steps to activate (needs prod / out-of-band) 1. **Apply** so the mirror timer populates the 4 packages (also picks up the merged #136 controller fix): ``` cd ansible && ansible-playbook site.yml --check --diff # preview ansible-playbook site.yml ``` Verified locally: `pnpm ansible:check` passes; `registry_mirror_images` has 5 entries on main. 2. **Mint a read-only token + set the Actions secret** (least-privilege — read:package only, distinct from the mirror's write token): ``` # create a read:package PAT on gitborg-ci, then: fj actions secrets set REGISTRY_READ_TOKEN --repo gitborg/gitborg-infra ``` 3. **Merge #139.** Anonymous pull is off (`git.gitborg.se/v2/` → 401) and the `gitborg` org must stay private, hence the login rather than public packages. Until 1–3 are done, smoke keeps pulling upstream (already resilient via the #138 retry).

supernaut refererade till detta ärende från en incheckning 2026-07-19 09:46:29 +00:00

fix(ci): smoke SKIPs (not FAILs) on a transient upstream registry outage (#137)

supernaut refererade till detta ärende från en incheckning 2026-07-19 09:51:13 +00:00

ci: best-effort registry login so smoke pulls from the mirror (#137) (#139)

supernaut kommenterad 2026-07-19 10:39:34 +00:00

Upphovsperson

Ägare

Kopiera länk

Step 1 done + verified (applied 2026-07-19)

site.yml --tags registry-mirror,runner-controller applied; health-gate green. The mirror now holds all four base images (pushed by gitborg-ci, verified via the packages API):

• git.gitborg.se/gitborg/forgejo:16.0.0
• git.gitborg.se/gitborg/postgres:17.10-trixie
• git.gitborg.se/gitborg/caddy:2.11.4-alpine
• git.gitborg.se/gitborg/kanidm:1.10.4

Both code PRs are merged: #138 (mirror-first pull + fallback + retry + transient-SKIP) and #139 (best-effort CI login).

Step 2 — remaining (only you can do this part)

The CI login needs a read-only PAT on the gitborg-ci service account — I can't mint another account's credential. Once it exists:

# create a read:package PAT on gitborg-ci, then:
fj actions secrets set REGISTRY_READ_TOKEN --repo gitborg/gitborg-infra

After that, the smoke job authenticates and pulls the base images from the mirror ([mirror]) instead of upstream. Until then it falls back to upstream (already resilient via retry + transient-SKIP).

When you've set the secret, ping me and I'll confirm a smoke run shows [mirror]. This issue stays open until then.

## Step 1 done + verified (applied 2026-07-19) `site.yml --tags registry-mirror,runner-controller` applied; health-gate green. The mirror now holds all four base images (pushed by `gitborg-ci`, verified via the packages API): - `git.gitborg.se/gitborg/forgejo:16.0.0` - `git.gitborg.se/gitborg/postgres:17.10-trixie` - `git.gitborg.se/gitborg/caddy:2.11.4-alpine` - `git.gitborg.se/gitborg/kanidm:1.10.4` Both code PRs are merged: #138 (mirror-first pull + fallback + retry + transient-SKIP) and #139 (best-effort CI login). ## Step 2 — remaining (only you can do this part) The CI login needs a **read-only PAT on the `gitborg-ci` service account** — I can't mint another account's credential. Once it exists: ``` # create a read:package PAT on gitborg-ci, then: fj actions secrets set REGISTRY_READ_TOKEN --repo gitborg/gitborg-infra ``` After that, the smoke job authenticates and pulls the base images from the mirror (`[mirror]`) instead of upstream. Until then it falls back to upstream (already resilient via retry + transient-SKIP). **When you've set the secret, ping me and I'll confirm a smoke run shows `[mirror]`.** This issue stays open until then.

supernaut refererade till detta ärende från en incheckning 2026-07-19 11:41:38 +00:00

ci: smoke registry login uses gitborg-bot, not gitborg-ci (#137)

supernaut refererade till detta ärende 2026-07-19 11:41:55 +00:00

ci: smoke registry login uses gitborg-bot, not gitborg-ci (#137) #140

supernaut refererade till detta ärende från en incheckning 2026-07-19 11:49:01 +00:00

ci: smoke registry login uses gitborg-bot, not gitborg-ci (#137) (#140)

supernaut kommenterad 2026-07-19 12:10:45 +00:00

Upphovsperson

Ägare

Kopiera länk

Done — closing.

Both steps complete and verified:

Step 1 (applied 2026-07-19): the four base images are mirrored into the Forgejo registry (bitborg/{forgejo:16.0.0,postgres:17.10-trixie,caddy:2.11.4-alpine,kanidm:1.10.4}), verified via the packages API.

Step 2:

• read:package PAT minted on gitborg-bot (least-privilege catch-all; gitborg-ci stays isolated to the deploy path), via an admin bot PAT — not a personal account.
• A direct GET /v2/gitborg/forgejo/manifests/16.0.0 as gitborg-bot returned 200 → the credential + org package-read both work.
• REGISTRY_READ_TOKEN set as the repo Actions secret.
• #140 merged → ci.yml logs in as gitborg-bot.

Code merged along the way: #138 (mirror-first pull + fallback + retry + transient-SKIP), #139 (best-effort login), #140 (login user → gitborg-bot).

Net: the smoke now pulls base images from the in-instance mirror, with upstream fallback + retry + transient-SKIP as the safety net — external-registry outages can no longer fail CI, and nothing is fetched from outside the instance on the happy path. The live [mirror] label will show on the next infra PR's smoke run.

## Done — closing. Both steps complete and verified: **Step 1 (applied 2026-07-19):** the four base images are mirrored into the Forgejo registry (`bitborg/{forgejo:16.0.0,postgres:17.10-trixie,caddy:2.11.4-alpine,kanidm:1.10.4}`), verified via the packages API. **Step 2:** - `read:package` PAT minted on **`gitborg-bot`** (least-privilege catch-all; `gitborg-ci` stays isolated to the deploy path), via an **admin bot** PAT — not a personal account. - A direct `GET /v2/gitborg/forgejo/manifests/16.0.0` as `gitborg-bot` returned **200** → the credential + org package-read both work. - `REGISTRY_READ_TOKEN` set as the repo Actions secret. - **#140** merged → `ci.yml` logs in as `gitborg-bot`. Code merged along the way: #138 (mirror-first pull + fallback + retry + transient-SKIP), #139 (best-effort login), #140 (login user → gitborg-bot). Net: the smoke now pulls base images from the in-instance mirror, with upstream fallback + retry + transient-SKIP as the safety net — external-registry outages can no longer fail CI, and nothing is fetched from outside the instance on the happy path. The live `[mirror]` label will show on the next infra PR's smoke run.

supernaut refererade till detta ärende 2026-07-19 12:13:18 +00:00

docs(runbook): mint service-account PATs via an admin bot, not a personal account #141

supernaut refererade till detta ärende från en incheckning 2026-07-19 12:16:30 +00:00

docs(runbook): mint service-account PATs via an admin bot, not a personal account (#141)

supernaut refererade till detta ärende från en incheckning 2026-07-19 14:18:00 +00:00

fix(backup-drill): free restore intermediates + bump drill VM to 80G

supernaut refererade till detta ärende 2026-07-19 14:20:02 +00:00

fix(backup-drill): free restore intermediates + bump drill VM to 80G #144

supernaut refererade till detta ärende från en incheckning 2026-07-19 14:30:24 +00:00

fix(backup-drill): free restore intermediates + bump drill VM to 80G (#144)

supernaut refererade till detta ärende 2026-07-19 15:04:47 +00:00

Restore drill fails with ENOSPC — drill VM boot volume (20 GB) < restored dataset (~25 GB) #131

supernaut refererade till detta ärende från en incheckning 2026-08-03 09:41:34 +00:00

fix(ci): smoke-containers pulls base images mirror-first with fallback + retry (#137) (#138)

supernaut refererade till detta ärende från en incheckning 2026-08-03 09:41:34 +00:00

ci: best-effort registry login so smoke pulls from the mirror (#137) (#139)

supernaut refererade till detta ärende från en incheckning 2026-08-03 09:41:34 +00:00

ci: smoke registry login uses bitborg-bot, not bitborg-ci (#137) (#140)

supernaut refererade till detta ärende från en incheckning 2026-08-03 09:41:34 +00:00

docs(runbook): mint service-account PATs via an admin bot, not a personal account (#141)

supernaut refererade till detta ärende från en incheckning 2026-08-03 09:41:34 +00:00

fix(backup-drill): free restore intermediates + bump drill VM to 80G (#144)

supernaut refererade till detta ärende 2026-08-18 18:53:40 +00:00

ci: validate the vmalert rule set before it can reach production #440

supernaut refererade till detta ärende från en incheckning 2026-08-18 19:15:38 +00:00

ci: validate the vmalert rule set before it can reach production (#440)

supernaut refererade till detta ärende 2026-08-18 19:58:22 +00:00

registry-mirror: the Kanidm pin drifted to 1.10.4 and nothing can notice #441

supernaut refererade till detta ärende från en incheckning 2026-10-02 17:56:57 +00:00

ci: validate the vmalert rule set before it can reach production (#440)

Logga in för att delta i denna konversation.

Ingen gren/etikett specificerad

Grenar Taggar

main

fix/log-retention-and-erasure

feat/billing-reconcile-trigger

chore/renovate-script-pins

Inga resultat hittades.

Etiketter

Rensa etiketter

  

area/backups

Backup & restore

  

area/ci

Actions / runners

  

area/control-panel

Logged-in self-service area

  

area/identity

Kanidm / OIDC / entitlements

  

area/infra

Infrastructure (OpenTofu/Ansible/host)

  

area/observability

Monitoring / logs / analytics

  

area/payments

Billing / payment providers

  

area/security

Security & compliance (incl. ISO 27001)

  

area/storage

Volumes / data / LFS

  

area/web

Public web portal (gitborg-web)

  

blocked

Evaluated and specified, but waiting on an external dependency

  

needs-info

Waiting on the reporter for more information

  

needs-triage

Not yet evaluated

  

ready-for-implementation

Fully specified, ready to be picked up

  

type

bug

Something is broken

  

type

chore

Maintenance / housekeeping

  

type

docs

Documentation

  

type

epic

High-level roadmap item

  

type

feature

New capability

  

type

task

Concrete unit of work

  

wontfix

Will not be actioned

Inga etiketter

area/backups

area/ci

area/control-panel

area/identity

area/infra

area/observability

area/payments

area/security

area/storage

area/web

blocked

needs-info

needs-triage

ready-for-implementation

type

bug

type

chore

type

docs

type

epic

type

feature

type

task

wontfix

Milstolpe

Rensa milstolpe

Inga objekt

Ingen milstolpe

Projekt

Rensa projekt

Inga objekt

Inget projekt

Bitborg Infrastructure

Tilldelade

Rensa tilldelade

Inga tilldelade

1 deltagare

Notiser

Prenumerera

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra#137

Ingen beskrivning angiven.