bitborg/bitborg-infra
Bevaka
2
0
Avgrena
Du har redan avgrenat bitborg-infra
 0
Kod Ärenden 64 Ändringsförfrågningar 3 Projekt 1 Utgåvor Paket Wiki Aktiviteter Actions

fail2ban caddy-auth: flip 401 deny-list to a credential-endpoint allow-list #196

Nytt ärende
Stängd
öppnade 2026-07-21 22:07:14 +00:00 av supernaut · 1 kommentar
supernaut kommenterad 2026-07-21 22:07:14 +00:00
Ägare
Kopiera länk

Problem

The caddy-auth fail2ban jail bans on any repeated HTTP 401, with a growing ignoreregex exclusion list for endpoints that answer 401 by protocol (not by wrong credential). This deny-list design has now broken CI twice:

  • #127/#175 → #179: the container-registry /v2/ handshake 401s banned the runner mid-push.
  • #195: the Forgejo Actions runner gRPC protocol /api/actions/runner.v1.RunnerService/* 401s banned the shared runner egress IP → runners couldn't declare → boot-loop → Action run 133 stuck 'waiting' for hours.

Every new endpoint that returns 401-by-protocol is a latent CI outage waiting to happen. Whack-a-mole exclusions don't scale.

Proposal

Invert the filter: instead of ban all 401s except an exclusion list, only count 401s on genuine credential-auth endpoints (git-over-HTTPS Basic auth paths, /user/login, /api/v1 token auth). Protocol handshakes (/v2/, /api/actions/, and any future ones) then never trip the jail without needing per-endpoint exclusions.

This is security-sensitive: the allow-list must enumerate the real brute-forceable surfaces carefully so we don't weaken brute-force protection. Verify with fail2ban-regex against a live access-log sample (both directions: real 401s still matched, protocol 401s still ignored) before deploying.

Also

  • RunnerQueueStalled fired only as warning and lagged the stall onset by ~3h — review its severity/for/routing so a runner boot-loop pages promptly instead of running silently.
  • Consider a pre-promotion smoke test of the runner image + a controller-side guard that flags 'runners boot but never declare (agent_labels null / never online)' as a distinct alert from plain queue depth.
## Problem The `caddy-auth` fail2ban jail bans on **any** repeated HTTP 401, with a growing `ignoreregex` exclusion list for endpoints that answer 401 *by protocol* (not by wrong credential). This deny-list design has now broken CI **twice**: - **#127/#175 → #179**: the container-registry `/v2/` handshake 401s banned the runner mid-push. - **#195**: the Forgejo Actions runner gRPC protocol `/api/actions/runner.v1.RunnerService/*` 401s banned the shared runner egress IP → runners couldn't declare → boot-loop → Action run 133 stuck 'waiting' for hours. Every new endpoint that returns 401-by-protocol is a latent CI outage waiting to happen. Whack-a-mole exclusions don't scale. ## Proposal Invert the filter: instead of *ban all 401s except an exclusion list*, **only count 401s on genuine credential-auth endpoints** (git-over-HTTPS Basic auth paths, `/user/login`, `/api/v1` token auth). Protocol handshakes (`/v2/`, `/api/actions/`, and any future ones) then never trip the jail without needing per-endpoint exclusions. This is security-sensitive: the allow-list must enumerate the real brute-forceable surfaces carefully so we don't *weaken* brute-force protection. Verify with `fail2ban-regex` against a live access-log sample (both directions: real 401s still matched, protocol 401s still ignored) before deploying. ## Also - `RunnerQueueStalled` fired only as **warning** and lagged the stall onset by ~3h — review its severity/`for`/routing so a runner boot-loop pages promptly instead of running silently. - Consider a pre-promotion smoke test of the runner image + a controller-side guard that flags 'runners boot but never declare (agent_labels null / never online)' as a distinct alert from plain queue depth.
supernaut kommenterad 2026-07-22 11:11:04 +00:00
Upphovsperson
Ägare
Kopiera länk

Follow-up complete: the RunnerQueueStalled severity/lag review (noted at close) shipped in PR #206 — added RunnerQueueStalledCritical (queued_jobs > 0 for: 30m, severity critical → pages), applied to the monitoring host + verified. Data confirmed the existing warning tier fired correctly ~15 min into the sustained stall; the only gap was that warning doesn't page, now covered by the 30-min critical escalation. No separate issue was tracked for this small follow-up.

Follow-up complete: the RunnerQueueStalled severity/lag review (noted at close) shipped in **PR #206** — added `RunnerQueueStalledCritical` (`queued_jobs > 0` `for: 30m`, severity **critical** → pages), applied to the monitoring host + verified. Data confirmed the existing warning tier fired correctly ~15 min into the sustained stall; the only gap was that warning doesn't page, now covered by the 30-min critical escalation. No separate issue was tracked for this small follow-up.
Logga in för att delta i denna konversation.
Ingen gren/etikett specificerad
Grenar Taggar
main
fix/log-retention-and-erasure
feat/billing-reconcile-trigger
chore/renovate-script-pins
Inga resultat hittades.
Etiketter
Rensa etiketter
  
area/backups
Backup & restore

  
area/ci
Actions / runners

  
area/control-panel
Logged-in self-service area

  
area/identity
Kanidm / OIDC / entitlements

  
area/infra
Infrastructure (OpenTofu/Ansible/host)

  
area/observability
Monitoring / logs / analytics

  
area/payments
Billing / payment providers

  
area/security
Security & compliance (incl. ISO 27001)

  
area/storage
Volumes / data / LFS

  
area/web
Public web portal (gitborg-web)

  
blocked
Evaluated and specified, but waiting on an external dependency

  
needs-info
Waiting on the reporter for more information

  
needs-triage
Not yet evaluated

  
ready-for-implementation
Fully specified, ready to be picked up

  
type
bug
 Something is broken

  
type
chore
 Maintenance / housekeeping

  
type
docs
 Documentation

  
type
epic
 High-level roadmap item

  
type
feature
 New capability

  
type
task
 Concrete unit of work

  
wontfix
Will not be actioned

Inga etiketter
area/backups
area/ci
area/control-panel
area/identity
area/infra
area/observability
area/payments
area/security
area/storage
area/web
blocked
needs-info
needs-triage
ready-for-implementation
type
bug
type
chore
type
docs
type
epic
type
feature
type
task
wontfix
Milstolpe
Rensa milstolpe
Inga objekt
Ingen milstolpe
Projekt
Rensa projekt
Inga objekt
Inget projekt
Bitborg Infrastructure
Tilldelade
Rensa tilldelade
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens
bitborg/bitborg-infra#196
Ingen beskrivning angiven.