fix(caddy): keep NET_BIND_SERVICE so the file-cap binary can exec (#81 hotfix) #102

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/caddy-net-bind-service-filecap in i main

2026-07-18 14:57:32 +00:00

supernaut kommenterad

2026-07-18 14:52:36 +00:00

Ägare

Hotfix for a prod outage caused by #94 (already applied to prod; this lands the fix on main so it isn't lost / re-broken on the next apply).

What happened

Applying #94 (socket activation) crash-looped Caddy with exec container process /usr/bin/caddy: Operation not permitted, taking all four hosts down for ~4 min (14:47–14:51 UTC, 2026-07-18).

Root cause

#94 dropped AddCapability=NET_BIND_SERVICE on the (correct) theory that with socket activation Caddy no longer binds ports. But the official caddy image ships /usr/bin/caddy with cap_net_bind_service as a file capability, and with NoNewPrivileges=true the kernel refuses to execve a file-capability binary unless that cap is in the bounding set → EPERM → crash loop. The dropped cap was never about binding; it's required just to exec the image's binary.

Fix

Restore AddCapability=NET_BIND_SERVICE (with a comment explaining it's for the exec, not for binding). PublishPort stays gone — the #81 win is intact; sockets are still host-bound and passed as fds.

Verified on prod (post-fix apply)

Container up; all 4 hosts serve (200/303/301, HTTP/2).
Real client IPs in both Caddy access log and Forgejo logs (was 10.89.0.x).
alt-svc: h3=":443" advertised; QUIC listener up (only a non-fatal UDP-buffer info note).
probe_success=1 for all endpoints; no new firing alerts.

Follow-up for reviewers

The lesson generalises: any Quadlet with NoNewPrivileges=true + DropCapability=ALL fronting an image whose binary carries file caps needs the matching AddCapability. Worth a note in the caddy/ADR-0005 hardening rationale.

**Hotfix for a prod outage caused by #94** (already applied to prod; this lands the fix on `main` so it isn't lost / re-broken on the next apply). ## What happened Applying #94 (socket activation) crash-looped Caddy with `exec container process /usr/bin/caddy: Operation not permitted`, taking all four hosts down for ~4 min (14:47–14:51 UTC, 2026-07-18). ## Root cause #94 dropped `AddCapability=NET_BIND_SERVICE` on the (correct) theory that with socket activation Caddy no longer *binds* ports. But the official `caddy` image ships `/usr/bin/caddy` with `cap_net_bind_service` as a **file capability**, and with `NoNewPrivileges=true` the kernel refuses to `execve` a file-capability binary unless that cap is in the bounding set → `EPERM` → crash loop. The dropped cap was never about binding; it's required just to exec the image's binary. ## Fix Restore `AddCapability=NET_BIND_SERVICE` (with a comment explaining it's for the exec, not for binding). `PublishPort` stays gone — the #81 win is intact; sockets are still host-bound and passed as fds. ## Verified on prod (post-fix apply) - Container up; all 4 hosts serve (200/303/301, HTTP/2). - **Real client IPs** in both Caddy access log and Forgejo logs (was `10.89.0.x`). - `alt-svc: h3=":443"` advertised; QUIC listener up (only a non-fatal UDP-buffer info note). - `probe_success=1` for all endpoints; no new firing alerts. ## Follow-up for reviewers The lesson generalises: any Quadlet with `NoNewPrivileges=true` + `DropCapability=ALL` fronting an image whose binary carries file caps needs the matching `AddCapability`. Worth a note in the caddy/ADR-0005 hardening rationale.

supernaut lade till 1 incheckning

2026-07-18 14:52:36 +00:00

fix(caddy): keep NET_BIND_SERVICE so the file-cap binary can exec (#81 hotfix)

ci / ci (pull_request) Successful in 2m6s

Detaljer

c77f2f3ea7

The #81 socket-activation change dropped AddCapability=NET_BIND_SERVICE
on the theory that socket activation means Caddy no longer binds ports.
But the official caddy image ships /usr/bin/caddy with cap_net_bind_service
as a FILE capability, and with NoNewPrivileges=true the kernel refuses to
execve a file-capability binary unless that cap is in the bounding set —
so the container crash-looped with 'exec /usr/bin/caddy: Operation not
permitted' and took prod down on apply (2026-07-18, ~4 min outage).

Restore the cap purely to satisfy the exec (it is NOT used to bind — the
sockets are still host-bound and passed as fds; PublishPort stays gone).

Verified on prod: container up, all 4 hosts serving, real client IPs in
caddy + forgejo logs, alt-svc h3 advertised, probe_success=1 all endpoints.

supernaut sammanfogade incheckning cb6a453a1a till main

2026-07-18 14:57:32 +00:00

supernaut tog bort grenen fix/caddy-net-bind-service-filecap

2026-07-18 14:57:32 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-18 14:57:33 +00:00

fix(caddy): keep NET_BIND_SERVICE so the file-cap binary can exec (#81 hotfix) (#102)

supernaut refererade till denna ändringsförfrågan

2026-07-18 16:18:15 +00:00

fix(token-audit): make token-audit.prom world-readable + group metric families (#75) #108

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-18 16:18:16 +00:00