fix(caddy,kanidm): edge rate limits for git and auth, no org creation by tier #296

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/launch-p0-hardening in i main

2026-08-01 08:50:41 +00:00

supernaut kommenterad

2026-08-01 00:06:44 +00:00

Ägare

Already applied to production and verified — this PR is the code catching up to the host, so
please merge promptly rather than treating it as pending work.

What changed

ADR 0032 Phase 2 rate limits on git. and auth. Phase 1's zone covers only the www block
and three sign-up paths, so both other hosts had no edge limit at all. Kanidm is the only sign-in
path (ENABLE_INTERNAL_SIGNIN=false), so an unthrottled auth host locks people out of Forgejo too;
and LANDING_PAGE=explore puts anonymous traffic straight onto the unthrottled Forgejo host.

git transport and the registry /v2/ are exempt by construction, per the ADR.

The auth zone is 300/min, not the ADR's ~20/min. Kanidm serves its SPA assets and several XHRs
from that host, so one genuine interactive login is well over 20 requests — a literal reading would
lock real users out mid-sign-in, on the only sign-in path there is. Flagging this as a deliberate
deviation for you to overrule if you disagree; the ADR does mandate tuning from data.

Internal callers exempt — this is the interesting part. The first version of the rate limits
broke the reconciler within six minutes. It reaches Forgejo over the public URL
(reconciler_forgejo_url = https://git.gitborg.se), so all of its calls arrived at Caddy under one
{remote_host} key, exceeded 120/min, and it aborted a reconcile with HTTP 429. Entitlement
enforcement, broken by a limit meant for strangers. Every zone now excludes private_ranges, which
narrows nothing for real users because they always arrive from public addresses.

Worth remembering as a pattern: this is the same shape as the fail2ban shared-SNAT incident (#195) —
first-party automation sharing one source identity with everything else behind it.

No tier grants organisation creation. An organisation is a priced add-on whose purchase flow
does not exist, so nesting a tier here gave it away free. It is also uncapped (Forgejo has no
per-user org limit) and each org is a separate 5 GiB quota owner, so any account with the entitlement
could mint orgs until the shared data volume filled — taking Forgejo, PostgreSQL and Kanidm down
together, since all three share it. That is the per-user cap bypass #121 closed, reachable again.

Emptying the member list alone did not work, and this is worth knowing generally: provisioning
is additive (overwriteMembers: false, --no-auto-remove), so a name dropped from the declared list
stays in the live group forever. The state template now takes a per-group overwrite_members flag,
enabled only for ent_orgs. That is safe there because every member is a tier group declared in
code — doing it blanket would wipe tier_* and ent_renovate, whose members the portal writes at
runtime, destroying every user's tier assignment and Renovate opt-in.

Verification in production

A reconcile completes with no HTTP 429 and gitborg_reconciler_last_run_status = 0.
org_create=false for every user, including alexanderkjall, kofish and bitborg-test-1, which
were true before. Quota (paid) and actions=true unchanged.
All six endpoints probe_success = 1; only Watchdog firing. ReconcilerFailed cleared.
All four rate-limit zones present in the running Caddyfile; caddy validate gated the restart.

Not in this PR

The LFS volume was grown 20→60 GB (/srv/gitborg-lfs was at 79% with 4 GB free, ~7 days from full)
and is now at 26% with 42 GB free. That change lives in terraform.tfvars, which is gitignored, so
it cannot appear here — flagging it so the tfvars drift is known rather than surprising.

Follow-ups worth filing

The registry has no retention policy. bitborg-web/cache had accumulated 44 versions from
--cache-to; the volume refills at ~0.6 GB/day. Growing it bought runway, it did not fix the leak.
resize2fs is manual. The filesystem task creates but never resizes, so a grown volume
silently does not grow its filesystem. resizefs: true would close that.
Nothing prunes UI-created Grafana dashboards — disableDeletion: false only removes dashboards
whose JSON file went away, so a hand-made one survives every converge.

**Already applied to production and verified** — this PR is the code catching up to the host, so please merge promptly rather than treating it as pending work. ## What changed **ADR 0032 Phase 2 rate limits on `git.` and `auth.`** Phase 1's zone covers only the `www` block and three sign-up paths, so both other hosts had no edge limit at all. Kanidm is the only sign-in path (`ENABLE_INTERNAL_SIGNIN=false`), so an unthrottled auth host locks people out of Forgejo too; and `LANDING_PAGE=explore` puts anonymous traffic straight onto the unthrottled Forgejo host. git transport and the registry `/v2/` are exempt by construction, per the ADR. **The auth zone is 300/min, not the ADR's ~20/min.** Kanidm serves its SPA assets and several XHRs from that host, so one genuine interactive login is well over 20 requests — a literal reading would lock real users out mid-sign-in, on the only sign-in path there is. Flagging this as a deliberate deviation for you to overrule if you disagree; the ADR does mandate tuning from data. **Internal callers exempt — this is the interesting part.** The first version of the rate limits broke the reconciler within six minutes. It reaches Forgejo over the *public* URL (`reconciler_forgejo_url = https://git.gitborg.se`), so all of its calls arrived at Caddy under one `{remote_host}` key, exceeded 120/min, and it aborted a reconcile with `HTTP 429`. Entitlement enforcement, broken by a limit meant for strangers. Every zone now excludes `private_ranges`, which narrows nothing for real users because they always arrive from public addresses. Worth remembering as a pattern: this is the same shape as the fail2ban shared-SNAT incident (#195) — first-party automation sharing one source identity with everything else behind it. **No tier grants organisation creation.** An organisation is a priced add-on whose purchase flow does not exist, so nesting a tier here gave it away free. It is also uncapped (Forgejo has no per-user org limit) and each org is a separate 5 GiB quota owner, so any account with the entitlement could mint orgs until the shared data volume filled — taking Forgejo, PostgreSQL and Kanidm down together, since all three share it. That is the per-user cap bypass #121 closed, reachable again. Emptying the member list alone did **not** work, and this is worth knowing generally: provisioning is additive (`overwriteMembers: false`, `--no-auto-remove`), so a name dropped from the declared list stays in the live group forever. The state template now takes a per-group `overwrite_members` flag, enabled **only** for `ent_orgs`. That is safe there because every member is a tier group declared in code — doing it blanket would wipe `tier_*` and `ent_renovate`, whose members the portal writes at runtime, destroying every user's tier assignment and Renovate opt-in. ## Verification in production - A reconcile completes with no `HTTP 429` and `gitborg_reconciler_last_run_status = 0`. - `org_create=false` for every user, including `alexanderkjall`, `kofish` and `bitborg-test-1`, which were `true` before. Quota (`paid`) and `actions=true` unchanged. - All six endpoints `probe_success = 1`; only `Watchdog` firing. `ReconcilerFailed` cleared. - All four rate-limit zones present in the running Caddyfile; `caddy validate` gated the restart. ## Not in this PR The LFS volume was grown 20→60 GB (`/srv/gitborg-lfs` was at 79% with 4 GB free, ~7 days from full) and is now at 26% with 42 GB free. That change lives in `terraform.tfvars`, which is gitignored, so it cannot appear here — flagging it so the tfvars drift is known rather than surprising. ## Follow-ups worth filing - **The registry has no retention policy.** `bitborg-web/cache` had accumulated 44 versions from `--cache-to`; the volume refills at ~0.6 GB/day. Growing it bought runway, it did not fix the leak. - **`resize2fs` is manual.** The `filesystem` task creates but never resizes, so a grown volume silently does not grow its filesystem. `resizefs: true` would close that. - **Nothing prunes UI-created Grafana dashboards** — `disableDeletion: false` only removes dashboards whose JSON file went away, so a hand-made one survives every converge.

supernaut lade till 1 incheckning

2026-08-01 00:06:44 +00:00

fix(caddy,kanidm): edge rate limits for git and auth, no org creation by tier

ci / ci (pull_request) Successful in 1m27s

Detaljer

82a0684ab3

Two launch-hardening changes found while reviewing exposure ahead of a public
announcement, plus the regression the first one caused.

**ADR 0032 Phase 2 rate limits.** Phase 1's zone is scoped to the www block and
three sign-up paths, so git.gitborg.se and auth.gitborg.se had no edge limit at
all. That mattered more than it looks: Kanidm is the only sign-in path
(ENABLE_INTERNAL_SIGNIN=false), so an unthrottled auth host means nobody can
reach Forgejo either, and LANDING_PAGE=explore puts anonymous traffic straight
onto the unthrottled Forgejo host.

git transport and the registry are exempt by construction, per the ADR: one
clone, push or pull is many sub-requests and CI arrives from a single NAT'd IP.

The auth zone is 300/min rather than the ADR's ~20/min. Kanidm serves its SPA
assets and several XHRs from that host, so one real interactive login exceeds 20
requests; a literal reading would have locked users out mid-sign-in. The ADR
mandates tuning from data, so revisit with Loki 429s.

**Internal callers are exempt, learned the hard way.** The first version of this
throttled the reconciler into failure within six minutes. It talks to Forgejo
over the public URL (reconciler_forgejo_url), so all of its calls arrived at
Caddy under one {remote_host} key, blew through 120/min, and it aborted a
reconcile with HTTP 429 — entitlement enforcement broken by a limit intended for
strangers. Every zone now excludes private_ranges, which narrows nothing for
real users since they always arrive from public addresses.

**No tier grants organisation creation.** An organisation is a priced add-on and
the purchase flow does not exist, so handing it out with a tier gave it away
free. It is also uncapped — Forgejo has no per-user org limit — and each org is
a separate 5 GiB quota owner, so any account with the entitlement could mint orgs
until the shared data volume filled, taking Forgejo, PostgreSQL and Kanidm down
together. That is the per-user cap bypass #121 closed, reachable again.

Emptying the member list was not enough on its own: provisioning is additive, so
a name dropped from the declared list stayed in the live group. The state
template now supports a per-group overwrite_members flag, enabled only for
ent_orgs — safe there because every member is a tier group declared in code,
unlike tier_* and ent_renovate, whose members are written at runtime by the
portal and would be wiped.

Verified in production: a reconcile now completes with no 429, org_create=false
for every user including three that were true, and quota and Actions unchanged.

supernaut sammanfogade incheckning e9ce4c56d9 till main

2026-08-01 08:50:41 +00:00

supernaut tog bort grenen fix/launch-p0-hardening

2026-08-01 08:50:41 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-01 08:50:43 +00:00

fix(caddy,kanidm): edge rate limits for git and auth, no org creation by tier (#296)

supernaut refererade till denna ändringsförfrågan

2026-08-01 09:03:03 +00:00

caddy: tune ADR 0032 rate limits from Loki 429 data, and revisit the auth zone #302

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-03 09:41:35 +00:00