Monitoring VM Caddy also loses real client IPs to pasta (grafana/stats/ntfy)

supernaut kommenterad

2026-07-18 14:28:09 +00:00

Ägare

Follow-up from #81 / PR #94. The services-host Caddy now sees real client IPs via systemd socket activation. The monitoring VM's own Caddy (fronting grafana.gitborg.se, stats.gitborg.se, ntfy.gitborg.se; roles/monitoring) is still a rootless Quadlet with published ports, so pasta rewrites inbound source addresses there too — its access logs (and any per-client logic) show the container-bridge address, not the real client.

Impact: lower than the main site (these are operator/admin surfaces, largely behind Kanidm SSO / basic auth), but:

Grafana/GoatCounter/ntfy access logs shipped to Loki show a bogus source IP.
Any future edge rate limiting or IP allowlisting on these hosts would treat all traffic as one client.
GoatCounter analytics on stats.gitborg.se would attribute all hits to one IP.

Fix: apply the same caddy.socket socket-activation pattern from #81/PR #94 to the monitoring role's Caddy (roles/monitoring/templates/caddy.container.j2 + a caddy.socket + the (socket_bind) Caddyfile snippet). The monitoring Caddy fronts three hosts on 80/443 — mirror the fd layout (fd/3 h1+h2, fdgram/4 h3, fd/5 http).

Verify after: systemctl --user status caddy.socket on the monitoring VM; a fresh external request shows a public remote_ip in the monitoring Caddy access log; GoatCounter records the real client.

Follow-up from #81 / PR #94. The services-host Caddy now sees real client IPs via systemd socket activation. The **monitoring VM's own Caddy** (fronting `grafana.gitborg.se`, `stats.gitborg.se`, `ntfy.gitborg.se`; `roles/monitoring`) is still a rootless Quadlet with published ports, so pasta rewrites inbound source addresses there too — its access logs (and any per-client logic) show the container-bridge address, not the real client. Impact: lower than the main site (these are operator/admin surfaces, largely behind Kanidm SSO / basic auth), but: - Grafana/GoatCounter/ntfy access logs shipped to Loki show a bogus source IP. - Any future edge rate limiting or IP allowlisting on these hosts would treat all traffic as one client. - GoatCounter analytics on `stats.gitborg.se` would attribute all hits to one IP. Fix: apply the same `caddy.socket` socket-activation pattern from #81/PR #94 to the monitoring role's Caddy (`roles/monitoring/templates/caddy.container.j2` + a `caddy.socket` + the `(socket_bind)` Caddyfile snippet). The monitoring Caddy fronts three hosts on 80/443 — mirror the fd layout (fd/3 h1+h2, fdgram/4 h3, fd/5 http). Verify after: `systemctl --user status caddy.socket` on the monitoring VM; a fresh external request shows a public `remote_ip` in the monitoring Caddy access log; GoatCounter records the real client.

supernaut lade till etiketterna

2026-07-18 14:28:32 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-18 14:36:10 +00:00

supernaut refererade till detta ärende

2026-07-18 14:36:37 +00:00

fix(caddy): socket-activate 80/443 so real client IPs reach caddy (#81) #94

supernaut refererade till detta ärende från en incheckning

2026-07-18 14:38:33 +00:00

fix(caddy): socket-activate 80/443 so real client IPs reach caddy (#81) (#94)

supernaut refererade till detta ärende

2026-07-19 06:58:04 +00:00

Caddy www block lacks trusted_proxies (strip client-supplied XFF at the edge) #129

supernaut refererade till detta ärende från en incheckning

2026-07-19 20:57:49 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-07-19 20:57:51 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs (#100) #154

supernaut refererade till detta ärende från en incheckning

2026-07-19 21:07:31 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs

supernaut refererade till detta ärende från en incheckning

2026-07-19 21:08:46 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs

supernaut stängde ärendet

2026-07-19 21:17:11 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-19 21:17:13 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs (#100) (#154)

supernaut kommenterad

2026-07-19 21:23:41 +00:00

Upphovsperson

Ägare

Reopening: PR #154 is merged but not yet applied — unlike bitborg-web (auto-deploys via CI→host-pull), this infra change is only live after ansible-playbook site.yml --tags monitoring, which restarts the monitoring Caddy onto the new socket-activated config. Until then the monitoring VM's Caddy still uses PublishPort (pasta) and logs the container-bridge IP, so the bug isn't actually fixed on prod.

Keeping this open until applied + verified on the monitoring VM: systemctl --user status caddy.socket active, grafana/stats/ntfy respond over HTTPS, and a fresh request shows a public remote_ip in the monitoring Caddy access log. Ready to apply on your go.

Reopening: PR #154 is **merged** but not yet **applied** — unlike bitborg-web (auto-deploys via CI→host-pull), this infra change is only live after `ansible-playbook site.yml --tags monitoring`, which restarts the monitoring Caddy onto the new socket-activated config. Until then the monitoring VM's Caddy still uses `PublishPort` (pasta) and logs the container-bridge IP, so the bug isn't actually fixed on prod. Keeping this open until applied + verified on the monitoring VM: `systemctl --user status caddy.socket` active, grafana/stats/ntfy respond over HTTPS, and a fresh request shows a public `remote_ip` in the monitoring Caddy access log. Ready to apply on your go.

supernaut återöppnade detta ärende

2026-07-19 21:24:29 +00:00

supernaut kommenterad

2026-07-19 21:33:00 +00:00

Upphovsperson

Ägare

✅ Applied + verified (PR #154, --tags monitoring, 2026-07-19).

First apply hit the expected EADDRINUSE at socket-start (the old Caddy still held 80/443 via pasta) — it failed safely (old Caddy stayed up, no outage). Stopped the old Caddy to free the ports, re-applied → clean (changed=3, 0 failed).

Verified on the monitoring VM:

caddy.socket active, caddy.service active.
Caddy container has zero published ports (podman port caddy empty) — pasta is entirely out of the inbound path, so nothing rewrites the source IP anymore. This is the identical socket-activation mechanism verified to deliver real client IPs on the services-host Caddy (#81).
All three hosts serve externally: grafana…/api/health → 200, ntfy…/v1/health → 200, stats… → 401 (GoatCounter behind basic-auth = up).

Note: a literal logged remote_ip couldn't be shown because the monitoring Caddyfile has no per-request access-log directive (unlike the services-host Caddy's (access_log) snippet). The IP-correctness follows structurally from pasta being removed. If you want the real IPs observable in Loki (and a literal confirmation), I can add an (access_log) snippet to the monitoring Caddy as a small follow-up.

✅ **Applied + verified** (PR #154, `--tags monitoring`, 2026-07-19). First apply hit the expected `EADDRINUSE` at socket-start (the old Caddy still held 80/443 via pasta) — it failed safely (old Caddy stayed up, no outage). Stopped the old Caddy to free the ports, re-applied → clean (changed=3, 0 failed). Verified on the monitoring VM: - `caddy.socket` **active**, `caddy.service` **active**. - Caddy container has **zero published ports** (`podman port caddy` empty) — pasta is entirely out of the inbound path, so nothing rewrites the source IP anymore. This is the identical socket-activation mechanism verified to deliver real client IPs on the services-host Caddy (#81). - All three hosts serve externally: `grafana…/api/health` → 200, `ntfy…/v1/health` → 200, `stats…` → 401 (GoatCounter behind basic-auth = up). Note: a literal logged remote_ip couldn't be shown because the monitoring Caddyfile has **no per-request access-log directive** (unlike the services-host Caddy's `(access_log)` snippet). The IP-correctness follows structurally from pasta being removed. If you want the real IPs observable in Loki (and a literal confirmation), I can add an `(access_log)` snippet to the monitoring Caddy as a small follow-up.

supernaut stängde ärendet

2026-07-19 21:33:01 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-19 21:37:33 +00:00

feat(monitoring): add access logging to the monitoring Caddy

supernaut refererade till detta ärende

2026-07-19 21:37:34 +00:00

feat(monitoring): access logging on the monitoring Caddy (real client IPs, #100) #156

supernaut kommenterad

2026-07-19 21:37:34 +00:00

Upphovsperson

Ägare

✅ Real client IPs confirmed live. Added access logging (PR follow-up) and external probes now log "remote_ip":"155.4.69.98" (+ other real public IPs like 155.4.244.250, 185.213.154.181) — zero pasta 10.x addresses. The socket-activation fix (#154) is delivering real client IPs; request attribution + GoatCounter analytics are correct.

✅ **Real client IPs confirmed live.** Added access logging (PR follow-up) and external probes now log `"remote_ip":"155.4.69.98"` (+ other real public IPs like 155.4.244.250, 185.213.154.181) — **zero pasta 10.x addresses**. The socket-activation fix (#154) is delivering real client IPs; request attribution + GoatCounter analytics are correct.

supernaut refererade till detta ärende från en incheckning

2026-07-19 21:41:16 +00:00

feat(monitoring): access logging on the monitoring Caddy (real client IPs, #100) (#156)

supernaut refererade till detta ärende från en incheckning

2026-08-03 09:41:33 +00:00

fix(caddy): socket-activate 80/443 so real client IPs reach caddy (#81) (#94)

supernaut refererade till detta ärende från en incheckning

2026-08-03 09:41:34 +00:00

fix(monitoring): socket-activate the monitoring Caddy for real client IPs (#100) (#154)

supernaut refererade till detta ärende från en incheckning

2026-08-03 09:41:34 +00:00

feat(monitoring): access logging on the monitoring Caddy (real client IPs, #100) (#156)