fail2ban jail for git-over-HTTPS / API 401 brute force #175

Sammanfogat

supernaut sammanfogade 1 incheckning från feat/127-fail2ban-https in i main

2026-07-20 18:19:33 +00:00

supernaut kommenterad

2026-07-20 15:18:11 +00:00

Ägare

Problem

Only [sshd] was jailed in jail.local.j2. HTTP Basic auth for git-over-HTTPS and
API-token auth stays on (ENABLE_INTERNAL_SIGNIN=false only removes the web sign-in form),
so those endpoints can be brute-forced with no host-layer lockout — nothing tailed Caddy's logs
for auth abuse.

Change

New filter roles/base/templates/filter.d/caddy-auth.conf.j2 — parses Caddy's JSON access
log for repeated HTTP 401 responses and extracts the client IP.
New jail [caddy-auth] in jail.local.j2, wired to the same nftables-multiport
banaction inherited from [DEFAULT] as [sshd] — no new ban mechanism.
Install mechanism: the base role now templates every templates/filter.d/*.conf.j2 into
/etc/fail2ban/filter.d/ (none existed before), before templating jail.local, both notifying
the existing Restart fail2ban handler. Add a filter by dropping in a new template.

Filter / jail design

Signal: "status":401. That is the answer to a bad Basic-auth credential or API token —
the actual brute-force surface. This covers /api/…, /user/login, and the git-HTTPS
…/info/refs + git-upload/receive-pack endpoints uniformly (a host-wide 401 match, as the
issue asked). 403 is deliberately excluded — Forgejo returns 403 for
authenticated-but-forbidden and for CSRF, which would ban legitimate logged-in users.
Real client IP: bans on request.remote_ip. Caddy is the edge and, via socket activation
(#81), sees the true external client as the TCP peer, logged as remote_ip — so we ban the
real client, not an internal proxy hop. (request.client_ip is the XFF-derived value; since
the only trusted proxy is the internal podman subnet, it equals remote_ip for real clients —
remote_ip is the unambiguous choice.) <ADDR> (IP-only) is used rather than <HOST> because
the field is always a bare IP; it matches both IPv4 and IPv6.
JSON timestamp: Caddy writes the time as a Unix epoch float in the top-level ts field.
fail2ban's bare EPOCH template only anchors on whitespace/line-start, which the "ts":
prefix defeats, so the datepattern embeds the epoch template behind that literal key:
datepattern = "ts":{EPOCH}. Parsing the real log time (not read time) is what stops a
fail2ban restart from re-scanning the file and mass-banning on stale 401s.
Log path: /home/gitborg/caddy/logs/access.log via fail2ban_caddy_log_path
({{ gitborg_home }}/caddy/logs/access.log) — the same host file the caddy role bind-mounts
and the monitoring-agent's Alloy already tails. Built from the global gitborg_home so the
base role carries no dependency on the caddy role's vars. The jail overrides the [DEFAULT]
systemd backend to polling because this source is a file, not the journal.

Tuning (all overridable in `roles/base/defaults/main.yml`)

param	value	rationale
maxretry	10	Looser than sshd's 3: git/CI clients legitimately retry a stale token a few times; 10×401 in 10 min is unambiguous brute force and won't lock out a fat-fingered token.
findtime	10m	Same window as the sshd jail.
bantime	1h	Shorter than sshd's 24h: HTTP creds typo/rotate more often and the per-try cost is already high behind TLS.
port	http,https	Scopes the nftables ban to the public web attack surface.

Verification

fail2ban-regex (v1.1.0) against five representative Caddy JSON lines
(IPv4 git-HTTPS 401, IPv4 /api/v1 401, IPv6 /user/login 401, a 200, a 403):

Failregex: 3 total
|   1) [3] ^.*?"remote_ip":"<ADDR>".*?"status":401(?:,|})
Date template hits:
|  [5] "ts":{EPOCH}
Lines: 5 lines, 0 ignored, 3 matched, 2 missed

The three 401s matched (IPv4 + IPv6 IPs extracted); the 200 and 403 were correctly missed; the
epoch parsed on all five lines.

Also green: pnpm ansible:check (syntax) and ansible-lint roles/base (production profile,
0 failures).

Deploy (maintainer)

Not applied to prod. Deploy is ansible-playbook site.yml --tags base; the Restart fail2ban
handler reloads the new jail + filter. On existing prod the access log already exists, so the
jail comes up immediately. Caveat: on a fresh provision base runs before caddy, so the
log file doesn't exist yet at first base converge — the [caddy-auth] jail is skipped until
the log appears; a second --tags base run (or the next full converge) settles it. [sshd] is
unaffected throughout.

Complementary to #48 (edge rate-limiting in Caddy): #48 throttles request rate at the
proxy; this bans credential-guessing clients at the host firewall. They stack.

Closes #127

## Problem Only `[sshd]` was jailed in `jail.local.j2`. HTTP Basic auth for **git-over-HTTPS** and **API-token** auth stays on (`ENABLE_INTERNAL_SIGNIN=false` only removes the web sign-in form), so those endpoints can be brute-forced with no host-layer lockout — nothing tailed Caddy's logs for auth abuse. ## Change - **New filter** `roles/base/templates/filter.d/caddy-auth.conf.j2` — parses Caddy's JSON access log for repeated **HTTP 401** responses and extracts the client IP. - **New jail** `[caddy-auth]` in `jail.local.j2`, wired to the **same** `nftables-multiport` banaction inherited from `[DEFAULT]` as `[sshd]` — no new ban mechanism. - **Install mechanism**: the base role now templates every `templates/filter.d/*.conf.j2` into `/etc/fail2ban/filter.d/` (none existed before), before templating `jail.local`, both notifying the existing `Restart fail2ban` handler. Add a filter by dropping in a new template. ## Filter / jail design - **Signal**: `"status":401`. That is the answer to a bad Basic-auth credential or API token — the actual brute-force surface. This covers `/api/…`, `/user/login`, and the git-HTTPS `…/info/refs` + `git-upload/receive-pack` endpoints uniformly (a host-wide 401 match, as the issue asked). **403 is deliberately excluded** — Forgejo returns 403 for authenticated-but-forbidden and for CSRF, which would ban legitimate logged-in users. - **Real client IP**: bans on `request.remote_ip`. Caddy is the edge and, via socket activation (#81), sees the true external client as the TCP peer, logged as `remote_ip` — so we ban the real client, not an internal proxy hop. (`request.client_ip` is the XFF-derived value; since the only trusted proxy is the internal podman subnet, it equals `remote_ip` for real clients — `remote_ip` is the unambiguous choice.) `<ADDR>` (IP-only) is used rather than `<HOST>` because the field is always a bare IP; it matches both IPv4 and IPv6. - **JSON timestamp**: Caddy writes the time as a Unix epoch float in the top-level `ts` field. fail2ban's bare `EPOCH` template only anchors on whitespace/line-start, which the `"ts":` prefix defeats, so the datepattern embeds the epoch template behind that literal key: `datepattern = "ts":{EPOCH}`. Parsing the *real* log time (not read time) is what stops a fail2ban restart from re-scanning the file and mass-banning on stale 401s. - **Log path**: `/home/gitborg/caddy/logs/access.log` via `fail2ban_caddy_log_path` (`{{ gitborg_home }}/caddy/logs/access.log`) — the same host file the caddy role bind-mounts and the monitoring-agent's Alloy already tails. Built from the global `gitborg_home` so the base role carries no dependency on the caddy role's vars. The jail overrides the `[DEFAULT]` `systemd` backend to `polling` because this source is a file, not the journal. ## Tuning (all overridable in `roles/base/defaults/main.yml`) | param | value | rationale | | -------- | ----- | --------- | | maxretry | 10 | Looser than sshd's 3: git/CI clients legitimately retry a stale token a few times; 10×401 in 10 min is unambiguous brute force and won't lock out a fat-fingered token. | | findtime | 10m | Same window as the sshd jail. | | bantime | 1h | Shorter than sshd's 24h: HTTP creds typo/rotate more often and the per-try cost is already high behind TLS. | | port | http,https | Scopes the nftables ban to the public web attack surface. | ## Verification `fail2ban-regex` (v1.1.0) against five representative Caddy JSON lines (IPv4 git-HTTPS 401, IPv4 `/api/v1` 401, IPv6 `/user/login` 401, a `200`, a `403`): ``` Failregex: 3 total | 1) [3] ^.*?"remote_ip":"<ADDR>".*?"status":401(?:,|}) Date template hits: | [5] "ts":{EPOCH} Lines: 5 lines, 0 ignored, 3 matched, 2 missed ``` The three 401s matched (IPv4 + IPv6 IPs extracted); the 200 and 403 were correctly missed; the epoch parsed on all five lines. Also green: `pnpm ansible:check` (syntax) and `ansible-lint roles/base` (production profile, 0 failures). ## Deploy (maintainer) Not applied to prod. Deploy is `ansible-playbook site.yml --tags base`; the `Restart fail2ban` handler reloads the new jail + filter. On existing prod the access log already exists, so the jail comes up immediately. **Caveat**: on a *fresh* provision `base` runs before `caddy`, so the log file doesn't exist yet at first `base` converge — the `[caddy-auth]` jail is skipped until the log appears; a second `--tags base` run (or the next full converge) settles it. `[sshd]` is unaffected throughout. ## Related Complementary to **#48** (edge rate-limiting in Caddy): #48 throttles request *rate* at the proxy; this bans *credential-guessing* clients at the host firewall. They stack. Closes #127

supernaut tvångsskickade feat/127-fail2ban-https från 49744649c1

ci / ci (pull_request) Successful in 2m22s

Detaljer

till 8200902b3a

ci / ci (pull_request) Successful in 1m32s

Detaljer

2026-07-20 18:07:36 +00:00

Jämför

supernaut sammanfogade incheckning ae915d7bc2 till main

2026-07-20 18:19:33 +00:00

supernaut tog bort grenen feat/127-fail2ban-https

2026-07-20 18:19:34 +00:00