bitborg/bitborg-infra
Bevaka
2
0
Avgrena
Du har redan avgrenat bitborg-infra
 0
Kod Ärenden 64 Ändringsförfrågningar 3 Projekt 1 Utgåvor Paket Wiki Aktiviteter Actions

SSH client IPs are rewritten by pasta (forgejo port 22 published) #91

Nytt ärende
Stängd
öppnade 2026-07-17 22:13:50 +00:00 av supernaut · 6 kommentarer
supernaut kommenterad 2026-07-17 22:13:50 +00:00
Ägare
Kopiera länk

Sibling of #81 (fixed for Caddy/HTTP via systemd socket activation): Forgejo's built-in SSH server has its port published through rootless Podman (pasta/rootlessport), so SSH connections reach Forgejo with a rewritten source address — SSH client IPs in Forgejo logs/audit trails are not real.

Impact: SSH-based abuse can't be attributed or rate-limited per client; audit logs for git-over-SSH show the container network address.

Fix candidates, in preference order:

  1. Socket activation for the forgejo Quadlet's SSH port — same pattern as #81's caddy.socket. Requires Forgejo's SSH listener to accept an inherited fd; Forgejo supports LISTEN_FDS-style activation for its web server via graceful restarts, but the built-in SSH server's fd-inheritance support needs verification first.
  2. PROXY protocol: front SSH with a small host-level forwarder that speaks PROXY protocol; Forgejo supports SSH_SERVER_PROXY_PROTOCOL (verify exact key/semantics in v16).
  3. Document as a known limitation if neither is practical.

Note the #81 verification steps in the runbook explicitly call this out as unresolved.

Sibling of #81 (fixed for Caddy/HTTP via systemd socket activation): Forgejo's built-in SSH server has its port published through rootless Podman (pasta/rootlessport), so SSH connections reach Forgejo with a rewritten source address — SSH client IPs in Forgejo logs/audit trails are not real. Impact: SSH-based abuse can't be attributed or rate-limited per client; audit logs for git-over-SSH show the container network address. Fix candidates, in preference order: 1. **Socket activation for the forgejo Quadlet's SSH port** — same pattern as #81's caddy.socket. Requires Forgejo's SSH listener to accept an inherited fd; Forgejo supports `LISTEN_FDS`-style activation for its web server via graceful restarts, but the built-in SSH server's fd-inheritance support needs verification first. 2. **PROXY protocol**: front SSH with a small host-level forwarder that speaks PROXY protocol; Forgejo supports `SSH_SERVER_PROXY_PROTOCOL` (verify exact key/semantics in v16). 3. Document as a known limitation if neither is practical. Note the #81 verification steps in the runbook explicitly call this out as unresolved.
supernaut kommenterad 2026-07-18 22:18:10 +00:00
Upphovsperson
Ägare
Kopiera länk

Analysis — the suggested fixes don't apply; the clean fix is ADR-level

Architecture: git-over-SSH is served by the forgejo image's bundled OpenSSH (app.ini START_SSH_SERVER=false — Forgejo's built-in Go SSH server is OFF; ADR 0006). It's published host:22 → container:22 via PublishPort (pasta), which SNATs the source to the container network — hence the lost client IP.

  • Socket activation (the #81 caddy fix) needs a listener that accepts an inherited fd. OpenSSH's main daemon has no LISTEN_FDS support (only per-connection sshd -i inetd mode). Doesn't drop in.
  • PROXY protocol (SSH_SERVER_PROXY_PROTOCOL) is a built-in-server option (disabled here); OpenSSH doesn't parse PROXY protocol. N/A.

The one clean fix: switch git-SSH to Forgejo's built-in server + socket-activate :22. The Go server can inherit an fd, and a systemd-bound host fd removes ADR 0006's "git user can't bind :22" objection and preserves the source IP. But this is an ADR-0006 revisit with real risk:

  • Host-key continuity: the built-in server uses its own host keys — unless the existing OpenSSH host keys are migrated, every user hits REMOTE HOST IDENTIFICATION HAS CHANGED. Must reuse the current keys.
  • The image starts sshd "regardless" → must disable it in the entrypoint.
  • Breaking git-SSH breaks all pushes/clones — exactly the "real-host-only, high-blast" class ADR 0030 says to rehearse first (Tier-1 / restore-drill clone).

Severity check: medium, attribution-only. fail2ban guards the admin sshd (2222), not git-SSH (:22), so no active per-IP control depends on the git-SSH source IP today. Git-over-HTTPS already logs real client IPs (#81) for anything needing attribution/rate-limiting. So the gap is forensic (can't tie git-SSH ops to real IPs in audit logs), not an enforcement hole.

Recommendation

Accept + document as a known limitation now, and schedule the built-in-server + socket-activation fix (with host-key migration) as a proper ADR + rehearsed change later. The fix's blast radius (all git-SSH + host-key continuity) isn't justified right now by an attribution-only gap that HTTPS already covers. If SSH-based abuse ever needs per-client action, revisit sooner.

### Analysis — the suggested fixes don't apply; the clean fix is ADR-level **Architecture:** git-over-SSH is served by the **forgejo image's bundled OpenSSH** (`app.ini` `START_SSH_SERVER=false` — Forgejo's built-in Go SSH server is OFF; ADR 0006). It's published `host:22 → container:22` via `PublishPort` (pasta), which SNATs the source to the container network — hence the lost client IP. - **Socket activation (the #81 caddy fix)** needs a listener that accepts an inherited fd. OpenSSH's main daemon has no `LISTEN_FDS` support (only per-connection `sshd -i` inetd mode). Doesn't drop in. - **PROXY protocol (`SSH_SERVER_PROXY_PROTOCOL`)** is a *built-in-server* option (disabled here); OpenSSH doesn't parse PROXY protocol. N/A. **The one clean fix:** switch git-SSH to **Forgejo's built-in server + socket-activate :22**. The Go server can inherit an fd, and a systemd-bound host fd removes ADR 0006's "git user can't bind :22" objection *and* preserves the source IP. But this is an **ADR-0006 revisit** with real risk: - **Host-key continuity:** the built-in server uses its own host keys — unless the existing OpenSSH host keys are migrated, every user hits `REMOTE HOST IDENTIFICATION HAS CHANGED`. Must reuse the current keys. - The image starts sshd "regardless" → must disable it in the entrypoint. - Breaking git-SSH breaks all pushes/clones — exactly the "real-host-only, high-blast" class ADR 0030 says to rehearse first (Tier-1 / restore-drill clone). **Severity check:** medium, attribution-only. fail2ban guards the *admin* sshd (2222), not git-SSH (:22), so **no active per-IP control depends on the git-SSH source IP** today. Git-over-HTTPS already logs real client IPs (#81) for anything needing attribution/rate-limiting. So the gap is forensic (can't tie git-SSH ops to real IPs in audit logs), not an enforcement hole. ### Recommendation **Accept + document as a known limitation now**, and schedule the built-in-server + socket-activation fix (with host-key migration) as a proper **ADR + rehearsed change** later. The fix's blast radius (all git-SSH + host-key continuity) isn't justified right now by an attribution-only gap that HTTPS already covers. If SSH-based abuse ever needs per-client action, revisit sooner.
supernaut kommenterad 2026-07-18 22:45:26 +00:00
Upphovsperson
Ägare
Kopiera länk

Approach decided: ADR 0031 (PR bitborg-docs#40) — option A2. Move git-SSH to the rootless Forgejo image's built-in server + systemd socket activation for :22 (the #81 pattern; Forgejo matches activated fds by address — verified). Feasibility confirmed; A1 rejected (rootful image starts openssh via s6 regardless).

Phased execution (rehearse-first, ADR 0030):

  1. Implement the rootless migration in the forgejo role + dependent roles (backup, backup-drill, reconciler/web exec-uid, GITEA_*/paths); validate + exercise in the local podman preview.
  2. Rehearse on a restore-drill clone (ADR 0027): boot from the latest backup, run the migrated site.yml, prove data intact + git clone/push over SSH + real client IP in Forgejo logs; nail the exact uid/path/userns volume-migration steps there.
  3. Prod cutover off-peak: fresh backup (rollback anchor) → apply → verify (#107 health gate + real SSH clone/push + real-IP check). Rollback = repin rootful image + restore the pre-cutover volume snapshot.

Now is the window (no external users → the uid/path/host-key migration is cheapest).

**Approach decided: ADR 0031** (PR bitborg-docs#40) — option A2. Move git-SSH to the rootless Forgejo image's built-in server + systemd socket activation for :22 (the #81 pattern; Forgejo matches activated fds by address — verified). Feasibility confirmed; A1 rejected (rootful image starts openssh via s6 regardless). **Phased execution (rehearse-first, ADR 0030):** 1. Implement the rootless migration in the `forgejo` role + dependent roles (backup, backup-drill, reconciler/web exec-uid, `GITEA_*`/paths); validate + exercise in the local podman preview. 2. **Rehearse on a restore-drill clone** (ADR 0027): boot from the latest backup, run the migrated `site.yml`, prove data intact + git clone/push over SSH + **real client IP** in Forgejo logs; nail the exact uid/path/userns volume-migration steps there. 3. **Prod cutover** off-peak: fresh backup (rollback anchor) → apply → verify (#107 health gate + real SSH clone/push + real-IP check). Rollback = repin rootful image + restore the pre-cutover volume snapshot. Now is the window (no external users → the uid/path/host-key migration is cheapest).
supernaut kommenterad 2026-07-18 23:03:00 +00:00
Upphovsperson
Ägare
Kopiera länk

Phase 1 (role changes) done — WIP draft PR #118 (feat/91-git-ssh-rootless), not applied (rehearsal-gated per ADR 0031).

Committed & syntax/lint-clean:

  • :16.0.0-rootless image; built-in SSH server (START_SSH_SERVER, SSH_LISTEN_HOST=::) claiming a passed socket fd.
  • new forgejo-ssh.socket (host :22, BindIPv6Only=both) → real client IP (#81-style).
  • forgejo.container: DropCapability=ALL, no PublishPort, Requires=forgejo-ssh.socket, rootless volume layout (/var/lib/gitea).
  • WORK_PATH=/var/lib/gitea; tasks socket-install + stop-before-rebind choreography.

Blocking remaining work (must run on the ADR 0031 rehearsal clone, not overnight/unattended):

  1. Data-volume migration script: rootful /data/gitea (uid 2000) → rootless /var/lib/gitea (uid 1000 subuid) — the copy + chown.
  2. Update backup-drill restore path for the new layout.
  3. Exec-verify on the clone: git clone+push over SSH, real client IP in logs, socket fd↔SSH_LISTEN_* address match, caps sufficient.
  4. Sync local/ + Makefile rootless assumptions.
  5. Off-peak prod cutover: backup anchor for rollback + the #107 health gate.
**Phase 1 (role changes) done** — WIP draft PR #118 (`feat/91-git-ssh-rootless`), **not applied** (rehearsal-gated per ADR 0031). Committed & syntax/lint-clean: - `:16.0.0-rootless` image; built-in SSH server (`START_SSH_SERVER`, `SSH_LISTEN_HOST=::`) claiming a passed socket fd. - new `forgejo-ssh.socket` (host :22, `BindIPv6Only=both`) → real client IP (#81-style). - `forgejo.container`: `DropCapability=ALL`, no `PublishPort`, `Requires=forgejo-ssh.socket`, rootless volume layout (`/var/lib/gitea`). - `WORK_PATH=/var/lib/gitea`; tasks socket-install + stop-before-rebind choreography. **Blocking remaining work (must run on the ADR 0031 rehearsal clone, not overnight/unattended):** 1. Data-volume migration script: rootful `/data/gitea` (uid 2000) → rootless `/var/lib/gitea` (uid 1000 subuid) — the copy + `chown`. 2. Update `backup-drill` restore path for the new layout. 3. Exec-verify on the clone: `git clone`+push over SSH, real client IP in logs, socket fd↔`SSH_LISTEN_*` address match, caps sufficient. 4. Sync `local/` + `Makefile` rootless assumptions. 5. Off-peak prod cutover: backup anchor for rollback + the #107 health gate.
supernaut kommenterad 2026-07-19 20:47:10 +00:00
Upphovsperson
Ägare
Kopiera länk

Investigation — the two fix candidates aren't feasible in the current architecture

Option 1 (socket activation, like #81): not applicable. #81 works because Caddy natively consumes a systemd-passed fd (bind fd/3 in the Caddyfile). git-over-SSH here is served by the image's bundled OpenSSH (START_SSH_SERVER=false; the rootful image runs its own sshd, PublishPort={{ git_ssh_port }}:22). OpenSSH sshd has no way to consume a systemd-passed socket fd for its main listener — so the fd-passing trick that de-pasta'd Caddy can't de-pasta sshd.

Option 2 (PROXY protocol): not applicable. SSH_SERVER_PROXY_PROTOCOL governs Forgejo's built-in SSH server, which is disabled. OpenSSH sshd doesn't speak PROXY protocol.

Resolution (option 3 for now) + recommendation

Documented as a known limitation in the runbook (this PR). A real fix requires a re-architecture of git-over-SSH, e.g.:

  • switch to Forgejo's built-in SSH server behind a host-level PROXY-protocol forwarder, or
  • a host-level sshd with AuthorizedKeysCommand proxying to Forgejo.

Both are sizeable changes to a core function (git push/pull) and warrant their own ADR/epic + a maintenance-window rollout — not an incidental fix. Impact is bounded: SSH access is key-authenticated and unaffected; only IP attribution in logs/audit is degraded (abuse-tracing / rate-limiting gap, not an access-control hole).

Keeping #91 open as the tracker for that re-architecture. Runbook now documents the limitation so it isn't rediscovered.

## Investigation — the two fix candidates aren't feasible in the current architecture **Option 1 (socket activation, like #81): not applicable.** #81 works because Caddy *natively consumes a systemd-passed fd* (`bind fd/3` in the Caddyfile). git-over-SSH here is served by the **image's bundled OpenSSH** (`START_SSH_SERVER=false`; the rootful image runs its own `sshd`, `PublishPort={{ git_ssh_port }}:22`). OpenSSH `sshd` has **no** way to consume a systemd-passed socket fd for its main listener — so the fd-passing trick that de-pasta'd Caddy can't de-pasta sshd. **Option 2 (PROXY protocol): not applicable.** `SSH_SERVER_PROXY_PROTOCOL` governs Forgejo's **built-in** SSH server, which is disabled. OpenSSH `sshd` doesn't speak PROXY protocol. ## Resolution (option 3 for now) + recommendation Documented as a known limitation in the runbook (this PR). A real fix requires a **re-architecture** of git-over-SSH, e.g.: - switch to Forgejo's built-in SSH server behind a host-level PROXY-protocol forwarder, or - a host-level `sshd` with `AuthorizedKeysCommand` proxying to Forgejo. Both are sizeable changes to a core function (git push/pull) and warrant their own ADR/epic + a maintenance-window rollout — not an incidental fix. **Impact is bounded:** SSH access is key-authenticated and unaffected; only IP *attribution* in logs/audit is degraded (abuse-tracing / rate-limiting gap, not an access-control hole). Keeping #91 open as the tracker for that re-architecture. Runbook now documents the limitation so it isn't rediscovered.
supernaut kommenterad 2026-07-19 22:58:31 +00:00
Upphovsperson
Ägare
Kopiera länk

Phase 1 complete — all remaining role work is done and validated in the local preview (commit 7e49b6e on feat/91-git-ssh-rootless; push + PR #118 refresh pending — SSH agent locked overnight).

Load-bearing discovery during validation: Forgejo does not read FORGEJO__* env natively (verified empirically on 16.0.0 — INSTALL_LOCK via env was ignored). Env config only works through environment-to-ini, which both images' entrypoints run unconditionally, rewriting $GITEA_APP_INI. Two consequences:

  1. On the rootful image this merge runs as container root (= host file owner), so prod's host-side forgejo-config/app.ini (mode 0644) has been getting the vaulted secrets persisted into it in plaintext on every container start — the known "permanent render diff" (2026-07-06 note in app.ini.j2) was this. On the rootless image (uid 1000, read-only mount) the same merge is simply fatal — caught in the local preview.
  2. Fix shipped in the unit: app.ini is mounted read-only at /etc/gitborg/app.ini, an entrypoint wrapper copies it to a tmpfs (/run/bitborg), and GITEA_APP_INI points there — env secrets merge in memory only. Host file pristine (verified), permanent render diff gone, and SECRET_KEY continuity across the cutover is guaranteed (the vaulted env values were and remain authoritative).

Local validation (podman, :16-rootless, --cap-drop=ALL): boots clean; admin created; git clone and push over the built-in SSH server OK; host app.ini contains no secrets after runs; branding mounts under custom/ OK.

Also in the commit: flag-gated volume migration (-e forgejo_migrate_rootless=true, podman unshare, gitea/*→root + chown 1000, leftovers stashed in .rootful-legacy/), layout-aware backup-drill restore (old + new archives), local/ synced (one-time make clean note in README), runbook end-state bullet.

Remaining (unchanged, user-gated): ADR 0031 phase 2 rehearsal on a restore-drill clone (data intact, SSH clone/push, real client IP, socket fd↔address match) → phase 3 off-peak prod cutover with backup anchor + #107 health gate.

**Phase 1 complete** — all remaining role work is done and validated in the local preview (commit `7e49b6e` on `feat/91-git-ssh-rootless`; push + PR #118 refresh pending — SSH agent locked overnight). **Load-bearing discovery during validation:** Forgejo does **not** read `FORGEJO__*` env natively (verified empirically on 16.0.0 — `INSTALL_LOCK` via env was ignored). Env config only works through `environment-to-ini`, which both images' entrypoints run **unconditionally, rewriting `$GITEA_APP_INI`**. Two consequences: 1. On the rootful image this merge runs as container root (= host file owner), so prod's host-side `forgejo-config/app.ini` (mode 0644) has been getting the vaulted secrets **persisted into it in plaintext on every container start** — the known "permanent render diff" (2026-07-06 note in app.ini.j2) was this. On the rootless image (uid 1000, read-only mount) the same merge is simply **fatal** — caught in the local preview. 2. Fix shipped in the unit: app.ini is mounted read-only at `/etc/gitborg/app.ini`, an entrypoint wrapper copies it to a tmpfs (`/run/bitborg`), and `GITEA_APP_INI` points there — env secrets merge **in memory only**. Host file pristine (verified), permanent render diff gone, and SECRET_KEY continuity across the cutover is guaranteed (the vaulted env values were and remain authoritative). **Local validation (podman, `:16-rootless`, `--cap-drop=ALL`):** boots clean; admin created; `git clone` **and push** over the built-in SSH server OK; host app.ini contains no secrets after runs; branding mounts under `custom/` OK. Also in the commit: flag-gated volume migration (`-e forgejo_migrate_rootless=true`, `podman unshare`, `gitea/*`→root + chown 1000, leftovers stashed in `.rootful-legacy/`), layout-aware backup-drill restore (old + new archives), `local/` synced (one-time `make clean` note in README), runbook end-state bullet. **Remaining (unchanged, user-gated):** ADR 0031 phase 2 rehearsal on a restore-drill clone (data intact, SSH clone/push, **real client IP**, socket fd↔address match) → phase 3 off-peak prod cutover with backup anchor + #107 health gate.
supernaut kommenterad 2026-07-24 19:29:13 +00:00
Upphovsperson
Ägare
Kopiera länk

Resolved by the ADR 0031 rootless migration (PR #118), cut over to prod 2026-07-24. git-SSH is now served by Forgejo's built-in SSH server on the :16.0.1-rootless image via forgejo-ssh.socket (systemd socket activation), so pasta no longer rewrites the source and real client IPs reach Forgejo. Verified in prod: SSH clone/push works and 'ss -tnp' shows the gitea process holding the :22 fd with the real public peer (no container-net address). A storage-uid regression during cutover (external [storage] still owned by the rootful uid 2000 vs the rootless image's fixed uid 1000) was fixed forward and at the root in 214b0c7. Note: Forgejo 16 does not log the SSH peer even at trace, so verify via ss / podman port, not podman logs (runbook updated).

Resolved by the ADR 0031 rootless migration (PR #118), cut over to prod 2026-07-24. git-SSH is now served by Forgejo's built-in SSH server on the :16.0.1-rootless image via forgejo-ssh.socket (systemd socket activation), so pasta no longer rewrites the source and real client IPs reach Forgejo. Verified in prod: SSH clone/push works and 'ss -tnp' shows the gitea process holding the :22 fd with the real public peer (no container-net address). A storage-uid regression during cutover (external [storage] still owned by the rootful uid 2000 vs the rootless image's fixed uid 1000) was fixed forward and at the root in 214b0c7. Note: Forgejo 16 does not log the SSH peer even at trace, so verify via ss / podman port, not podman logs (runbook updated).
supernaut 2026-07-24 19:29:13 +00:00
Logga in för att delta i denna konversation.
Ingen gren/etikett specificerad
Grenar Taggar
main
fix/log-retention-and-erasure
feat/billing-reconcile-trigger
chore/renovate-script-pins
Inga resultat hittades.
Etiketter
Rensa etiketter
  
area/backups
Backup & restore

  
area/ci
Actions / runners

  
area/control-panel
Logged-in self-service area

  
area/identity
Kanidm / OIDC / entitlements

  
area/infra
Infrastructure (OpenTofu/Ansible/host)

  
area/observability
Monitoring / logs / analytics

  
area/payments
Billing / payment providers

  
area/security
Security & compliance (incl. ISO 27001)

  
area/storage
Volumes / data / LFS

  
area/web
Public web portal (gitborg-web)

  
blocked
Evaluated and specified, but waiting on an external dependency

  
needs-info
Waiting on the reporter for more information

  
needs-triage
Not yet evaluated

  
ready-for-implementation
Fully specified, ready to be picked up

  
type
bug
 Something is broken

  
type
chore
 Maintenance / housekeeping

  
type
docs
 Documentation

  
type
epic
 High-level roadmap item

  
type
feature
 New capability

  
type
task
 Concrete unit of work

  
wontfix
Will not be actioned

Inga etiketter
area/backups
area/ci
area/control-panel
area/identity
area/infra
area/observability
area/payments
area/security
area/storage
area/web
blocked
needs-info
needs-triage
ready-for-implementation
type
bug
type
chore
type
docs
type
epic
type
feature
type
task
wontfix
Milstolpe
Rensa milstolpe
Inga objekt
Ingen milstolpe
Projekt
Rensa projekt
Inga objekt
Inget projekt
Bitborg Infrastructure
Tilldelade
Rensa tilldelade
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens
bitborg/bitborg-infra#91
Ingen beskrivning angiven.