git-SSH via rootless image + built-in SSH server (ADR 0031, #91) — rehearsed + prod cutover done #118

Sammanfogat

supernaut sammanfogade 24 incheckningar från feat/91-git-ssh-rootless in i main 2026-07-24 19:18:11 +00:00

Konversation 1 Incheckningar 24 Ändrade filer 16 +1253 -100

supernaut kommenterad 2026-07-18 23:00:32 +00:00

Ägare

Kopiera länk

✅ LANDED — prod cutover complete 2026-07-24. Forgejo is live on 16.0.1-rootless with git-SSH via socket activation. The ADR 0031 phase-2 rehearsal passed and the phase-3 prod cutover is done; the health gate is green and main now matches deployed prod. Safe to merge.

Phase 1 of #91 / ADR 0031. Grounded in the verified rootless image config (User=1000, GITEA_WORK_DIR=/var/lib/gitea, GITEA_CUSTOM=/var/lib/gitea/custom).

In the branch (phase 1):

• Image → :16.0.1-rootless; Forgejo's built-in SSH server (START_SSH_SERVER=true, SSH_LISTEN_HOST=:: / SSH_LISTEN_PORT matched to the socket by address).
• forgejo-ssh.socket (user unit) binds host :22 and passes the fd into the container → real client IP in logs (the #81 caddy pattern). forgejo.container drops OpenSSH (DropCapability=ALL, NoNewPrivileges=true), no PublishPort, Requires=forgejo-ssh.socket.
• app.ini mounted read-only + copied to a tmpfs (GITEA_APP_INI=/run/gitborg/app.ini) so the entrypoint's environment-to-ini merge never writes secrets to the host file.
• One-time data migration migrate-rootless.yml (gated on -e forgejo_migrate_rootless=true): podman unshare, gitea/* → volume root, chown uid 1000, rootful leftovers stashed in .rootful-legacy/.
• backup-drill restore script detects rootful-vs-rootless layout; local/Makefile + local vars synced.

Two cutover-blocking fixes (commit 9cd473b) — neither exercised by the local preview (it publishes -p 2222:2222 and never uses socket activation):

1. forgejo.container: removed a duplicate custom/{templates,public} Volume= mount (rebase artifact from the #188 merge) → podman "duplicate mount destination" → container won't start.
2. forgejo-ssh.socket: added Service=forgejo.service → without it the socket fd targets a non-existent forgejo-ssh.service, so git-over-SSH would be dead (no PublishPort fallback).

Phase-2 rehearsal tooling: so the rootful→rootless cutover could be rehearsed on a throwaway restore-drill clone before prod (ADR 0031 / 0027 / 0030).

• backup-drill: a manual DRILL_HOLD=1 mode that boots + data-stages a clone as the weekly drill does, but skips restore/doctor/teardown and holds the VM (with a shutdown self-destruct backstop) for a rehearsal. The weekly timer never sets DRILL_HOLD — automated drill behaviour is unchanged.
• ansible/rehearsal-rootless.yml: reaches the clone (FIP-direct, or ProxyJump via the services host), bootstraps the host preconditions the forgejo role needs (bitborg user, subuid/subgid, linger, :22 sysctl, quadlet dirs, aardvark-dns + the gitborg network, and an admin-sshd → :2222 cutover so Forgejo's git-SSH gets the real :22), restores prod data in rootful layout, then runs the real, unmodified forgejo role with forgejo_migrate_rootless=true and asserts every ADR 0031 §2 criterion plus a manual clone/push + real-client-IP checklist.
• opentofu/backup-drill.tofu: drill SG allows :2222 ingress (admin SSH after the clone's cutover).
• docs/runbook.md: the full rehearsal procedure + mandatory teardown (the clone materialises real prod signing secrets → destroy it after) + go/no-go gate.

Rehearsal (phase 2) — PASSED (2026-07-23, re-validated 2026-07-24 on a clone of live prod data): all ADR 0031 §2 asserts green (socket + service active, volume migrated rootful→rootless, caps dropped, forgejo doctor, secret-free host app.ini); git-SSH clone/push works; protected-main correctly rejected; real client IP confirmed via ss (the gitea process holds the :22 fd with the real public peer — no pasta rewrite). Note: Forgejo 16 does not console-log the SSH peer even at trace, so verification is via ss / podman port, not podman logs — runbook corrected.

Cutover (phase 3) — DONE (2026-07-24; fresh backup anchor → site.yml --tags forgejo -e forgejo_migrate_rootless=true). The migration succeeded, but Forgejo then crash-looped briefly on a storage-uid bug: the external [storage] volume (/srv/gitborg-lfs, mounted /srv/storage) was still owned by the rootful git uid (2000) while the rootless image's git user is a fixed 1000 → mkdir /srv/storage/attachments: permission denied. Fixed forward (recursive podman unshare chown 1000 + restart), and fixed at the root in commit 214b0c7: the steady-state storage-chown task now targets uid 1000, and migrate-rootless.yml gained a one-time recursive storage chown (the storage analogue of the data-volume chown). The rehearsal missed this because it skipped the external [storage] volume — now covered. The follow-up apply is green; the health gate (#107) confirms [storage] serves a DB-known object and git.gitborg.se is up.

Also in this branch (corrections captured during the cutover): a dpkg-lock wait in the rehearsal bootstrap (first-boot unattended-upgrades race), and runbook fixes — the manual DRILL_HOLD invocation environment, the --tags backup-drill prerequisite, and ss-based client-IP verification for both the rehearsal and the prod cutover.

pnpm ansible:check, the rehearsal playbook --syntax-check, ansible-lint (production profile), and tofu validate all clean.

Closes #207.

Refs #91.

✅ **LANDED — prod cutover complete 2026-07-24.** Forgejo is live on `16.0.1-rootless` with git-SSH via socket activation. The ADR 0031 phase-2 rehearsal passed and the phase-3 prod cutover is done; the health gate is green and `main` now matches deployed prod. Safe to merge. **Phase 1 of #91 / ADR 0031.** Grounded in the verified rootless image config (`User=1000`, `GITEA_WORK_DIR=/var/lib/gitea`, `GITEA_CUSTOM=/var/lib/gitea/custom`). **In the branch (phase 1):** - Image → `:16.0.1-rootless`; Forgejo's built-in SSH server (`START_SSH_SERVER=true`, `SSH_LISTEN_HOST=::` / `SSH_LISTEN_PORT` matched to the socket by address). - `forgejo-ssh.socket` (user unit) binds host :22 and passes the fd into the container → **real client IP** in logs (the #81 caddy pattern). `forgejo.container` drops OpenSSH (`DropCapability=ALL`, `NoNewPrivileges=true`), no `PublishPort`, `Requires=forgejo-ssh.socket`. - app.ini mounted read-only + copied to a tmpfs (`GITEA_APP_INI=/run/gitborg/app.ini`) so the entrypoint's `environment-to-ini` merge never writes secrets to the host file. - One-time data migration `migrate-rootless.yml` (gated on `-e forgejo_migrate_rootless=true`): `podman unshare`, `gitea/*` → volume root, chown uid 1000, rootful leftovers stashed in `.rootful-legacy/`. - backup-drill restore script detects rootful-vs-rootless layout; `local/Makefile` + local vars synced. **Two cutover-blocking fixes (commit `9cd473b`)** — neither exercised by the local preview (it publishes `-p 2222:2222` and never uses socket activation): 1. `forgejo.container`: removed a **duplicate** `custom/{templates,public}` `Volume=` mount (rebase artifact from the #188 merge) → podman "duplicate mount destination" → container won't start. 2. `forgejo-ssh.socket`: added `Service=forgejo.service` → without it the socket fd targets a non-existent `forgejo-ssh.service`, so git-over-SSH would be dead (no `PublishPort` fallback). **Phase-2 rehearsal tooling:** so the rootful→rootless cutover could be rehearsed on a throwaway restore-drill clone before prod (ADR 0031 / 0027 / 0030). - `backup-drill`: a manual **`DRILL_HOLD=1`** mode that boots + data-stages a clone as the weekly drill does, but skips restore/doctor/teardown and holds the VM (with a `shutdown` self-destruct backstop) for a rehearsal. The weekly timer never sets `DRILL_HOLD` — automated drill behaviour is unchanged. - `ansible/rehearsal-rootless.yml`: reaches the clone (FIP-direct, or ProxyJump via the services host), bootstraps the host preconditions the `forgejo` role needs (bitborg user, subuid/subgid, linger, `:22` sysctl, quadlet dirs, aardvark-dns + the `gitborg` network, and an admin-sshd → `:2222` cutover so Forgejo's git-SSH gets the real `:22`), restores prod data in **rootful layout**, then runs the **real, unmodified `forgejo` role with `forgejo_migrate_rootless=true`** and asserts every ADR 0031 §2 criterion plus a manual clone/push + real-client-IP checklist. - `opentofu/backup-drill.tofu`: drill SG allows `:2222` ingress (admin SSH after the clone's cutover). - `docs/runbook.md`: the full rehearsal procedure + **mandatory teardown** (the clone materialises real prod signing secrets → destroy it after) + go/no-go gate. **Rehearsal (phase 2) — PASSED** (2026-07-23, re-validated 2026-07-24 on a clone of live prod data): all ADR 0031 §2 asserts green (socket + service active, volume migrated rootful→rootless, caps dropped, `forgejo doctor`, secret-free host app.ini); git-SSH clone/push works; protected-`main` correctly rejected; **real client IP confirmed via `ss`** (the `gitea` process holds the `:22` fd with the real public peer — no pasta rewrite). Note: Forgejo 16 does not console-log the SSH peer even at `trace`, so verification is via `ss` / `podman port`, not `podman logs` — runbook corrected. **Cutover (phase 3) — DONE** (2026-07-24; fresh backup anchor → `site.yml --tags forgejo -e forgejo_migrate_rootless=true`). The migration succeeded, but Forgejo then crash-looped briefly on a **storage-uid bug**: the external `[storage]` volume (`/srv/gitborg-lfs`, mounted `/srv/storage`) was still owned by the rootful git uid (2000) while the rootless image's git user is a fixed 1000 → `mkdir /srv/storage/attachments: permission denied`. Fixed forward (recursive `podman unshare chown 1000` + restart), and fixed at the root in **commit `214b0c7`**: the steady-state storage-chown task now targets uid 1000, and `migrate-rootless.yml` gained a one-time recursive storage chown (the storage analogue of the data-volume chown). The rehearsal missed this because it skipped the external `[storage]` volume — now covered. The follow-up apply is green; the health gate (#107) confirms `[storage]` serves a DB-known object and `git.gitborg.se` is up. **Also in this branch** (corrections captured during the cutover): a dpkg-lock wait in the rehearsal bootstrap (first-boot `unattended-upgrades` race), and runbook fixes — the manual `DRILL_HOLD` invocation environment, the `--tags backup-drill` prerequisite, and `ss`-based client-IP verification for both the rehearsal and the prod cutover. `pnpm ansible:check`, the rehearsal playbook `--syntax-check`, `ansible-lint` (production profile), and `tofu validate` all clean. Closes #207. Refs #91.

supernaut refererade till denna ändringsförfrågan 2026-07-18 23:03:00 +00:00

SSH client IPs are rewritten by pasta (forgejo port 22 published) #91

supernaut tvångsskickade feat/91-git-ssh-rootless från ac6bc3d16a till b85bd6560b 2026-07-19 22:17:05 +00:00 Jämför

supernaut refererade till denna ändringsförfrågan 2026-07-19 22:58:31 +00:00

SSH client IPs are rewritten by pasta (forgejo port 22 published) #91

supernaut kommenterad 2026-07-20 11:35:20 +00:00

Upphovsperson

Ägare

Kopiera länk

Phase 1 complete + rebased on main (still rehearsal-gated)

The role changes are done and validated in the local podman preview, not just drafted. Branch head 7e49b6e.

Added since the original WIP push:

• tmpfs config merge — app.ini mounted read-only at /etc/gitborg/app.ini; an entrypoint wrapper copies it to a tmpfs and GITEA_APP_INI points there, so the image's environment-to-ini merges the env secrets in memory instead of rewriting the host file. Discovery: Forgejo does NOT read FORGEJO__* natively — that merge is the only mechanism, and on the rootful image it has been persisting vaulted secrets in plaintext into the host app.ini (the 2026-07-06 "permanent render diff"). Now fixed + gone; SECRET_KEY continuity across cutover is safe.
• Flag-gated data-volume migration (-e forgejo_migrate_rootless=true), layout-aware backup-drill restore, local/ synced, runbook end-state.

Local validation (:16-rootless, --cap-drop=ALL): boots clean, admin created, git clone and push over the built-in SSH server OK, host app.ini secret-free.

Remaining (user-gated): ADR 0031 phase 2 rehearsal on a restore-drill clone (real-client-IP proof) → phase 3 off-peak prod cutover. Do not apply until then.

### Phase 1 complete + rebased on main (still rehearsal-gated) The role changes are **done and validated in the local podman preview**, not just drafted. Branch head `7e49b6e`. **Added since the original WIP push:** - **tmpfs config merge** — app.ini mounted read-only at `/etc/gitborg/app.ini`; an entrypoint wrapper copies it to a tmpfs and `GITEA_APP_INI` points there, so the image's `environment-to-ini` merges the env secrets **in memory** instead of rewriting the host file. Discovery: Forgejo does NOT read `FORGEJO__*` natively — that merge is the only mechanism, and on the rootful image it has been persisting vaulted secrets in plaintext into the host app.ini (the 2026-07-06 "permanent render diff"). Now fixed + gone; SECRET_KEY continuity across cutover is safe. - Flag-gated data-volume migration (`-e forgejo_migrate_rootless=true`), layout-aware backup-drill restore, `local/` synced, runbook end-state. **Local validation** (`:16-rootless`, `--cap-drop=ALL`): boots clean, admin created, `git clone` **and push** over the built-in SSH server OK, host app.ini secret-free. **Remaining (user-gated):** ADR 0031 phase 2 rehearsal on a restore-drill clone (real-client-IP proof) → phase 3 off-peak prod cutover. **Do not apply until then.**

supernaut tvångsskickade feat/91-git-ssh-rootless från 7e49b6e54f till 7839ceeece 2026-07-20 18:09:42 +00:00 Jämför

supernaut tvångsskickade feat/91-git-ssh-rootless från 7839ceeece till ce7342c120 2026-07-21 11:33:03 +00:00 Jämför

supernaut tvångsskickade feat/91-git-ssh-rootless från ce7342c120 till 0abd5df64e 2026-07-21 19:30:15 +00:00 Jämför

supernaut tvångsskickade feat/91-git-ssh-rootless från 0abd5df64e till 10c0555364 2026-07-22 11:23:28 +00:00 Jämför

supernaut ändrade titeln från WIP: git-SSH via rootless image + built-in server (#91) — DO NOT APPLY (rehearsal-gated) till git-SSH via rootless image + built-in server + phase-2 rehearsal tooling (#91) — DRAFT, rehearsal-gated 2026-07-22 13:53:43 +00:00

supernaut ändrade titeln från git-SSH via rootless image + built-in server + phase-2 rehearsal tooling (#91) — DRAFT, rehearsal-gated till WIP: git-SSH via rootless image + built-in server + phase-2 rehearsal tooling (#91) — rehearsal-gated 2026-07-22 13:54:16 +00:00

supernaut tvångsskickade feat/91-git-ssh-rootless från 58258b6f82 till 9b1bab9603 2026-07-22 14:04:14 +00:00 Jämför

supernaut tvångsskickade feat/91-git-ssh-rootless från 9b1bab9603 till aaaad747e7 2026-07-22 15:05:00 +00:00 Jämför

supernaut tvångsskickade feat/91-git-ssh-rootless från aaaad747e7 till 02831d5e5f 2026-07-22 18:01:45 +00:00 Jämför

supernaut refererade till denna ändringsförfrågan 2026-07-22 18:16:35 +00:00

forgejo role: forgejo_work_path default is stale after the rootless migration (ADR 0031) #207

supernaut tvångsskickade feat/91-git-ssh-rootless från 59262df850 till 675b32ee3e 2026-07-24 15:51:24 +00:00 Jämför

supernaut lade till 1 incheckning 2026-07-24 18:31:24 +00:00

fix(rehearsal): dpkg-lock wait + runbook fixes from the 2026-07-24 re-rehearsal (#91) ... a5e8fa27d5

- rehearsal-rootless.yml: stop first-boot unattended-upgrades and wait for the
  dpkg frontend lock before the bootstrap apt install (it raced the lock and
  failed on a fresh clone).
- runbook: the DRILL_HOLD manual invocation needs the full drill-service env
  (absolute path, DRILL_HOLD inside env, OS_CLIENT_CONFIG_FILE, sourced
  backup.env) — a bare form silently runs the normal drill and tears down.
- runbook: new prerequisite — deploy the backup-drill role (--tags backup-drill)
  before HOLD mode works on prod.
- runbook: correct the real-client-IP check — Forgejo 16 does not log the SSH
  peer even at trace; verify via 'ss -tnp | grep :22' + 'podman port'.
- runbook: reachability notes (stale known_hosts on FIP reuse, cd /tmp for
  sudo -u gitborg, DBUS_SESSION_BUS_ADDRESS for manual podman).

supernaut lade till 2 incheckningar 2026-07-24 19:13:50 +00:00

docs(runbook): fix the prod SSH client-IP verification note too — ss, not podman logs (#91) e538ead8b8

fix(forgejo): chown the external [storage] volume to the rootless git uid 1000 (#91) ... 214b0c7122

The -rootless image's git user is a fixed uid 1000, not gitborg_user_uid
(2000, the rootful git uid). The storage-chown task in main.yml used 2000, so
after the ADR 0031 cutover the [storage] mount stayed owned by 2000 and
rootless Forgejo (uid 1000) crashed on startup: 'mkdir /srv/storage/
attachments: permission denied'. Fix the steady-state task to 1000, and add a
one-time recursive storage chown to migrate-rootless.yml (pruning lost+found)
so pre-existing rootful-era subdirs are converted during the cutover — the
storage analogue of the data-volume chown. The rehearsal missed this because it
skipped the external storage volume.

supernaut ändrade titeln från WIP: git-SSH via rootless image + built-in server + phase-2 rehearsal tooling (#91) — rehearsal-gated till git-SSH via rootless image + built-in SSH server (ADR 0031, #91) — rehearsed + prod cutover done 2026-07-24 19:16:35 +00:00

supernaut sammanfogade incheckning 4e214e7d95 till main 2026-07-24 19:18:11 +00:00

supernaut tog bort grenen feat/91-git-ssh-rootless 2026-07-24 19:18:11 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning 2026-07-24 19:18:11 +00:00

git-SSH via rootless image + built-in SSH server (ADR 0031, #91) — rehearsed + prod cutover done (#118)

supernaut refererade till denna ändringsförfrågan 2026-07-24 19:29:13 +00:00

SSH client IPs are rewritten by pasta (forgejo port 22 published) #91

supernaut refererade denna ändringsförfrågan från en incheckning 2026-08-03 09:41:34 +00:00

git-SSH via rootless image + built-in SSH server (ADR 0031, #91) — rehearsed + prod cutover done (#118)

Logga in för att delta i denna konversation.

Granskare

Inga granskare

Etiketter

Rensa etiketter

  

area/backups

Backup & restore

  

area/ci

Actions / runners

  

area/control-panel

Logged-in self-service area

  

area/identity

Kanidm / OIDC / entitlements

  

area/infra

Infrastructure (OpenTofu/Ansible/host)

  

area/observability

Monitoring / logs / analytics

  

area/payments

Billing / payment providers

  

area/security

Security & compliance (incl. ISO 27001)

  

area/storage

Volumes / data / LFS

  

area/web

Public web portal (gitborg-web)

  

blocked

Evaluated and specified, but waiting on an external dependency

  

needs-info

Waiting on the reporter for more information

  

needs-triage

Not yet evaluated

  

ready-for-implementation

Fully specified, ready to be picked up

  

type

bug

Something is broken

  

type

chore

Maintenance / housekeeping

  

type

docs

Documentation

  

type

epic

High-level roadmap item

  

type

feature

New capability

  

type

task

Concrete unit of work

  

wontfix

Will not be actioned

Inga etiketter

area/backups

area/ci

area/control-panel

area/identity

area/infra

area/observability

area/payments

area/security

area/storage

area/web

blocked

needs-info

needs-triage

ready-for-implementation

type

bug

type

chore

type

docs

type

epic

type

feature

type

task

wontfix

Milstolpe

Rensa milstolpe

Inga objekt

Ingen milstolpe

Projekt

Rensa projekt

Inga objekt

Inget projekt

Tilldelade

Rensa tilldelade

Inga tilldelade

1 deltagare

Notiser

Prenumerera

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra!118

Ingen beskrivning angiven.