Verify HTTP/3 (QUIC) over the socket-activated fd after applying #81

supernaut kommenterad

2026-07-18 14:28:25 +00:00

Ägare

Post-apply verification for #81 / PR #94. The socket-activation change binds HTTP/3 (QUIC) via ListenDatagram=443 in caddy.socket → bind fdgram/4 { protocols h3 } in the Caddyfile. This path could not be exercised in the local podman machine (no curl --http3 there), so it's verified only by config-validation, not end-to-end.

Once PR #94 is applied to prod, confirm HTTP/3 actually works over the inherited datagram fd:

Browser devtools (Network → Protocol column shows h3) against https://www.gitborg.se and https://git.gitborg.se, or curl --http3 -sI https://www.gitborg.se if a HTTP/3-capable curl is available.
Confirm the alt-svc: h3=":443" response header is advertised.
Check Caddy logs for no QUIC/datagram-fd errors on start.

Fallback if h3 misbehaves over the inherited fd: drop the ListenDatagram=443 line from caddy.socket and the bind fdgram/4 { protocols h3 } block from the (socket_bind) Caddyfile snippet — Caddy falls back to HTTP/1.1 + HTTP/2 on fd/3 with no other change. Track the outcome here (either close as verified, or open a fix if the fd path needs adjustment).

Post-apply verification for #81 / PR #94. The socket-activation change binds HTTP/3 (QUIC) via `ListenDatagram=443` in `caddy.socket` → `bind fdgram/4 { protocols h3 }` in the Caddyfile. This path could **not** be exercised in the local podman machine (no `curl --http3` there), so it's verified only by config-validation, not end-to-end. Once PR #94 is applied to prod, confirm HTTP/3 actually works over the inherited datagram fd: - [ ] Browser devtools (Network → Protocol column shows `h3`) against `https://www.gitborg.se` and `https://git.gitborg.se`, or `curl --http3 -sI https://www.gitborg.se` if a HTTP/3-capable curl is available. - [ ] Confirm the `alt-svc: h3=":443"` response header is advertised. - [ ] Check Caddy logs for no QUIC/datagram-fd errors on start. Fallback if h3 misbehaves over the inherited fd: drop the `ListenDatagram=443` line from `caddy.socket` and the `bind fdgram/4 { protocols h3 }` block from the `(socket_bind)` Caddyfile snippet — Caddy falls back to HTTP/1.1 + HTTP/2 on fd/3 with no other change. Track the outcome here (either close as verified, or open a fix if the fd path needs adjustment).

supernaut lade till etiketterna

2026-07-18 14:28:33 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-18 14:36:10 +00:00

supernaut refererade till detta ärende

2026-07-18 14:36:37 +00:00

fix(caddy): socket-activate 80/443 so real client IPs reach caddy (#81) #94

supernaut refererade till detta ärende från en incheckning

2026-07-18 14:38:33 +00:00

fix(caddy): socket-activate 80/443 so real client IPs reach caddy (#81) (#94)

supernaut kommenterad

2026-07-18 14:53:09 +00:00

Upphovsperson

Ägare

Partial verification after the #94 apply to prod (2026-07-18):

✅ alt-svc: h3=":443"; ma=2592000 advertised on https://www.gitborg.se.
✅ QUIC listener started on the inherited datagram fd (fdgram/4) — no bind/fd errors in Caddy logs.
⚠️ Caddy logs one non-fatal QUIC note: failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). HTTP/3 works, but throughput is capped until net.core.rmem_max/wmem_max are raised (https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes). Candidate: add the sysctl to the base/podman role.
⛔ Could not complete an end-to-end h3 client handshake: the control node's curl has no HTTP/3. Still needs a real h3 client (browser devtools Protocol column, or curl --http3) to fully close this.

Net: h3 is offered and the listener is healthy; remaining = (1) confirm a real h3 handshake, (2) decide on the UDP-buffer sysctl.

Partial verification after the #94 apply to prod (2026-07-18): - ✅ `alt-svc: h3=":443"; ma=2592000` advertised on https://www.gitborg.se. - ✅ QUIC listener started on the inherited datagram fd (fdgram/4) — no bind/fd errors in Caddy logs. - ⚠️ Caddy logs one **non-fatal** QUIC note: `failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB)`. HTTP/3 works, but throughput is capped until `net.core.rmem_max`/`wmem_max` are raised (https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes). Candidate: add the sysctl to the base/podman role. - ⛔ Could **not** complete an end-to-end h3 client handshake: the control node's curl has no HTTP/3. Still needs a real h3 client (browser devtools Protocol column, or `curl --http3`) to fully close this. Net: h3 is offered and the listener is healthy; remaining = (1) confirm a real h3 handshake, (2) decide on the UDP-buffer sysctl.

supernaut kommenterad

2026-07-18 21:50:59 +00:00

Upphovsperson

Ägare

Verified — HTTP/3 does not work on prod (root cause found)

Drove an HTTP/3-only client (curl --http3-only, badouralix/curl-http3 image) from a container against prod:

https://www.gitborg.se/   → http_version=0 code=000   (h3 connect failed)
https://git.gitborg.se/   → http_version=0 code=000
https://auth.gitborg.se/  → http_version=0 code=000

Control (same image/host): cloudflare-quic.com and google.com → http_version=3 code=200. So the client/egress is fine — prod's h3 is genuinely broken, not a test artifact.

Root cause: UDP/443 is not open at the perimeter. The ingress model is TCP-only:

roles/base/templates/nftables.conf.j2 renders tcp dport … from gitborg_ingress_tcp_ports;
opentofu/security.tofu renders the Neutron SG with protocol = "tcp" from the same list.

So inbound QUIC (UDP/443) is dropped at both the host firewall and the OpenStack SG. This predates #81 (the pre-#81 PublishPort=443/udp bound the container, but the perimeter still filtered UDP) — #81 carried forward a never-working h3. Net effect today: Caddy advertises alt-svc: h3=":443", clients attempt h3, fail, and fall back to h2 — a small connection-setup penalty for zero benefit.

Decision needed — enable or disable h3

A. Enable h3: extend the ingress model to carry UDP/443 (touches the ingress_tcp_ports shape + nftables template + security.tofu), then apply base + tofu (an SG change — handle carefully per infra-apply). Cost: a new public UDP port (attack surface), firewall + SG change.
B. Disable h3 (recommended): drop ListenDatagram=443 from caddy.socket + the bind fdgram/4 { protocols h3 } from the Caddyfile so Caddy stops advertising a filtered protocol — honest h1/h2-only. This is this issue's own documented fallback. Cheapest, least-privilege, and h2 is entirely sufficient for a git host + portal.

### Verified — HTTP/3 does not work on prod (root cause found) Drove an HTTP/3-only client (`curl --http3-only`, badouralix/curl-http3 image) from a container against prod: ``` https://www.gitborg.se/ → http_version=0 code=000 (h3 connect failed) https://git.gitborg.se/ → http_version=0 code=000 https://auth.gitborg.se/ → http_version=0 code=000 ``` Control (same image/host): `cloudflare-quic.com` and `google.com` → **http_version=3 code=200**. So the client/egress is fine — **prod's h3 is genuinely broken**, not a test artifact. **Root cause: UDP/443 is not open at the perimeter.** The ingress model is TCP-only: - `roles/base/templates/nftables.conf.j2` renders `tcp dport …` from `gitborg_ingress_tcp_ports`; - `opentofu/security.tofu` renders the Neutron SG with `protocol = "tcp"` from the same list. So inbound QUIC (UDP/443) is dropped at both the host firewall and the OpenStack SG. This predates #81 (the pre-#81 `PublishPort=443/udp` bound the container, but the perimeter still filtered UDP) — #81 carried forward a never-working h3. Net effect today: Caddy advertises `alt-svc: h3=":443"`, clients attempt h3, fail, and fall back to h2 — a small connection-setup penalty for zero benefit. ### Decision needed — enable or disable h3 - **A. Enable h3:** extend the ingress model to carry UDP/443 (touches the `ingress_tcp_ports` shape + nftables template + `security.tofu`), then apply base + tofu (an SG change — handle carefully per infra-apply). Cost: a new public UDP port (attack surface), firewall + SG change. - **B. Disable h3 (recommended):** drop `ListenDatagram=443` from `caddy.socket` + the `bind fdgram/4 { protocols h3 }` from the Caddyfile so Caddy stops advertising a filtered protocol — honest h1/h2-only. This is this issue's own documented fallback. Cheapest, least-privilege, and h2 is entirely sufficient for a git host + portal.

supernaut refererade till detta ärende från en incheckning

2026-07-18 21:59:00 +00:00

fix(caddy): stop advertising HTTP/3 — UDP/443 is filtered at the perimeter (#101)

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-07-18 21:59:17 +00:00

fix(caddy): stop advertising HTTP/3 — UDP/443 is filtered at the perimeter (#101) #117

supernaut stängde ärendet

2026-07-18 22:07:08 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-18 22:07:08 +00:00

fix(caddy): stop advertising HTTP/3 — UDP/443 is filtered at the perimeter (#101) (#117)

supernaut refererade till detta ärende från en incheckning