bitborg/bitborg-infra
Bevaka
2
0
Avgrena
Du har redan avgrenat bitborg-infra
 0
Kod Ärenden 64 Ändringsförfrågningar 3 Projekt 1 Utgåvor Paket Wiki Aktiviteter Actions

security: production Forgejo is 20 days behind an arbitrary-file-read fix (16.0.2) #447

Nytt ärende
Stängd
öppnade 2026-08-19 06:56:31 +00:00 av supernaut · 1 kommentar
supernaut kommenterad 2026-08-19 06:56:31 +00:00
Ägare
Kopiera länk

Forgejo 16.0.2 is a security release. Production runs 16.0.1-rootless, shipped behind it since
2026-07-30.

The fix that matters

fix: prevent arbitrary file reads from org-mode rendering (upstream PR 13682).

Org-mode rendering happens when a .org file is viewed. Reaching it needs a repository holding an
attacker-crafted file, which needs an account, and the portal is the only route to an account.

The consequence is not contained to the file being read. Anything readable by the Forgejo process
includes app.ini, which holds SECRET_KEY, INTERNAL_TOKEN, the database credentials, the mailer
credentials and the OIDC client secret. SECRET_KEY and INTERNAL_TOKEN are what forge sessions and
decrypt stored values, so a read of that one file is an escalation rather than an information leak.

The same release also carries:

  • fix: add CGNAT (100.64.0.0/10) to 'private' hostmatcher rule (PR 13547/13567). The private
    hostmatcher backs the webhook and migration allowlists, so an SSRF target in that range was not
    being treated as internal.
  • fix: prevent stored XSS in Actions pre-execution warnings and errors (PR 13682). Needs Actions
    access, which is entitlement-gated, so this is the least reachable of the three.
  • Eight dependency bumps upstream marked [SECURITY]: klauspost/compress, go-chi/chi, postcss,
    golang.org/x/text, google.golang.org/grpc, svgo, and others.

Why it sat unshipped for 20 days

Two defects compounded, and neither logged an error.

  1. #430. minimumReleaseAgeBehaviour defaults to timestamp-required and the docker datasource
    returned no releaseTimestamp, so a check that needed a timestamp could never pass. The update sat
    in the dashboard's "Pending Status Checks" with an approvePr-branch action and no branch, for
    seventeen days.
  2. #446. With that fixed, the update moved to "Awaiting Schedule" and inherited the global
    before 6am on monday window. vulnerabilityAlerts exempts security work from the schedule, but
    only for updates Renovate itself classifies, and there is no vulnerability feed for a container
    image tag. So the security release queued behind a weekly window like an ordinary bump.

Worth recording what did not fail. scripts/check-renovate-annotations.py reported forgejo as
watched throughout, correctly: the annotation was present, the datasource resolved, and Renovate saw
the new version every night. The dependency was watched, tracked and listed, and nobody was told.
Coverage is not delivery.

Sequence

  • [~] Sign-up closure prepared and then deliberately SKIPPED. PR #448 was written and CI-green,
    then closed unmerged. It was insurance against the concealment CSS being broken by 16.0.2,
    which would have made the upgrade slow. Verification passed, so the upgrade was unblocked
    immediately and closing sign-up would have cost two extra applies and two portal restarts to
    protect a window of minutes. The branch is kept, not deleted, so it is one merge away if the
    apply is ever delayed.
  • Concealment surfaces verified against 16.0.2 in the local/ preview rather than on prod.
    local/Makefile pins 16-rootless, a floating minor, so it pulls 16.0.2 today. ADR 0038's gate
    exists because CSS concealment no-ops silently when upstream restructures a settings page.
  • forgejo_image_tag → 16.0.2-rootless and health_check_forgejo_concealment_verified_tag
    → 16.0.2-rootless in ONE change    , per the gate's own instruction. Never pre-bump the verified
    tag, never commit a disabled gate.
  • Applied to bitborg-prod, Forgejo restarted, running image confirmed as 16.0.2 rather than
    assumed. Note #63: an image-tag bump can no-op without a pull and restart.
  • Sign-up reopened (web_signups_open: "true") in the same session the tag bump lands.
  • Any concealment CSS broken by 16.0.2 fixed before sign-up reopens, not after.

The registry mirror needs no separate change: registry_mirror_images templates forgejo's entry from
{{ forgejo_image_tag }}, so the mirror follows the bump. That is not true of kanidm, which is a
hardcoded literal (#441).

Follow-up this exposes

#446 was filed as a structural worst case of six days. It is now evidenced as a real 25-day exposure,
which should raise its priority rather than leave it in triage.

Verification result, 2026-08-19

Done in the local/ preview (which needed repairing first: it had been broken since the ADR 0039
rename, fixed in #449). Preview confirmed running forgejo version 16.0.2+gitea-1.22.0.

Selector Matches on 16.0.2 State With bitborg.css disabled
label:has(input[name="full_name"]) 1 HIDDEN VISIBLE
.right.floated.content:has(input[name="_method"][value="PRIMARY"]) 1 HIDDEN VISIBLE

Both selectors still match 16.0.2 markup and still hide. The toggle is what makes that meaningful:
"nothing visible" is also what a selector matching nothing produces. The PRIMARY row needed a second
email address added first, because with a single address there is no primary to choose, the selector
matched zero elements, and it would have passed vacuously.

Signed-out navbar also checked: "Create account" present, pointing at www.bitborg.se/en/signup.

Residual, NOT closed by this

EXTERNAL_USER_DISABLE_FEATURES = deletion,manage_password applies only to external OIDC users, so the
preview's break-glass local admin cannot exercise it. If 16.0.2 renamed the key or changed its accepted
values, external users silently regain password management and self-delete. Needs one real OIDC account
checked after the apply. It is config rather than CSS, so the concealment gate does not cover it either.

This also explains a false alarm worth recording: the preview shows a "Change password" section and a
"Delete your account" heading. That is correct for a local admin, and the config comment says so
outright. It is not the concealment failing.

Upgrade PR

#450, with the dry-run accounted for task by task. Two of the six changed tasks are the tag propagating
through templated consumers (the backup-drill orchestrator and the registry-mirror wrapper), so the
mirror carries 16.0.2 and Saturday's restore drill runs forgejo doctor against it, with no second
literal to chase.

Forgejo 16.0.2 is a security release. Production runs 16.0.1-rootless, shipped behind it since 2026-07-30. ## The fix that matters **`fix: prevent arbitrary file reads from org-mode rendering`** (upstream PR 13682). Org-mode rendering happens when a `.org` file is viewed. Reaching it needs a repository holding an attacker-crafted file, which needs an account, and the portal is the only route to an account. The consequence is not contained to the file being read. Anything readable by the Forgejo process includes `app.ini`, which holds `SECRET_KEY`, `INTERNAL_TOKEN`, the database credentials, the mailer credentials and the OIDC client secret. `SECRET_KEY` and `INTERNAL_TOKEN` are what forge sessions and decrypt stored values, so a read of that one file is an escalation rather than an information leak. The same release also carries: - `fix: add CGNAT (100.64.0.0/10) to 'private' hostmatcher rule` (PR 13547/13567). The `private` hostmatcher backs the webhook and migration allowlists, so an SSRF target in that range was not being treated as internal. - `fix: prevent stored XSS in Actions pre-execution warnings and errors` (PR 13682). Needs Actions access, which is entitlement-gated, so this is the least reachable of the three. - Eight dependency bumps upstream marked `[SECURITY]`: `klauspost/compress`, `go-chi/chi`, `postcss`, `golang.org/x/text`, `google.golang.org/grpc`, `svgo`, and others. ## Why it sat unshipped for 20 days Two defects compounded, and neither logged an error. 1. **#430.** `minimumReleaseAgeBehaviour` defaults to `timestamp-required` and the docker datasource returned no `releaseTimestamp`, so a check that needed a timestamp could never pass. The update sat in the dashboard's "Pending Status Checks" with an `approvePr-branch` action and no branch, for seventeen days. 2. **#446.** With that fixed, the update moved to "Awaiting Schedule" and inherited the global `before 6am on monday` window. `vulnerabilityAlerts` exempts security work from the schedule, but only for updates Renovate itself classifies, and there is no vulnerability feed for a container image tag. So the security release queued behind a weekly window like an ordinary bump. Worth recording what did **not** fail. `scripts/check-renovate-annotations.py` reported forgejo as watched throughout, correctly: the annotation was present, the datasource resolved, and Renovate saw the new version every night. The dependency was watched, tracked and listed, and nobody was told. Coverage is not delivery. ## Sequence - [~] **Sign-up closure prepared and then deliberately SKIPPED.** PR #448 was written and CI-green, then closed unmerged. It was insurance against the concealment CSS being broken by 16.0.2, which would have made the upgrade slow. Verification passed, so the upgrade was unblocked immediately and closing sign-up would have cost two extra applies and two portal restarts to protect a window of minutes. The branch is kept, not deleted, so it is one merge away if the apply is ever delayed. - [x] **Concealment surfaces verified against 16.0.2** in the `local/` preview rather than on prod. `local/Makefile` pins `16-rootless`, a floating minor, so it pulls 16.0.2 today. ADR 0038's gate exists because CSS concealment no-ops silently when upstream restructures a settings page. - [x] **`forgejo_image_tag` → `16.0.2-rootless` and `health_check_forgejo_concealment_verified_tag` → `16.0.2-rootless` in ONE change**, per the gate's own instruction. Never pre-bump the verified tag, never commit a disabled gate. - [ ] **Applied to bitborg-prod**, Forgejo restarted, running image confirmed as 16.0.2 rather than assumed. Note #63: an image-tag bump can no-op without a pull and restart. - [ ] **Sign-up reopened** (`web_signups_open: "true"`) in the same session the tag bump lands. - [ ] Any concealment CSS broken by 16.0.2 fixed before sign-up reopens, not after. The registry mirror needs no separate change: `registry_mirror_images` templates forgejo's entry from `{{ forgejo_image_tag }}`, so the mirror follows the bump. That is not true of kanidm, which is a hardcoded literal (#441). ## Follow-up this exposes #446 was filed as a structural worst case of six days. It is now evidenced as a real 25-day exposure, which should raise its priority rather than leave it in triage. ## Verification result, 2026-08-19 Done in the `local/` preview (which needed repairing first: it had been broken since the ADR 0039 rename, fixed in #449). Preview confirmed running `forgejo version 16.0.2+gitea-1.22.0`. | Selector | Matches on 16.0.2 | State | With `bitborg.css` disabled | | --- | --- | --- | --- | | `label:has(input[name="full_name"])` | 1 | HIDDEN | **VISIBLE** | | `.right.floated.content:has(input[name="_method"][value="PRIMARY"])` | 1 | HIDDEN | **VISIBLE** | Both selectors still match 16.0.2 markup and still hide. The toggle is what makes that meaningful: "nothing visible" is also what a selector matching nothing produces. The PRIMARY row needed a second email address added first, because with a single address there is no primary to choose, the selector matched zero elements, and it would have passed vacuously. Signed-out navbar also checked: "Create account" present, pointing at `www.bitborg.se/en/signup`. ### Residual, NOT closed by this `EXTERNAL_USER_DISABLE_FEATURES = deletion,manage_password` applies only to external OIDC users, so the preview's break-glass local admin cannot exercise it. If 16.0.2 renamed the key or changed its accepted values, external users silently regain password management and self-delete. Needs one real OIDC account checked after the apply. It is config rather than CSS, so the concealment gate does not cover it either. This also explains a false alarm worth recording: the preview shows a "Change password" section and a "Delete your account" heading. That is **correct** for a local admin, and the config comment says so outright. It is not the concealment failing. ## Upgrade PR #450, with the dry-run accounted for task by task. Two of the six changed tasks are the tag propagating through templated consumers (the backup-drill orchestrator and the registry-mirror wrapper), so the mirror carries 16.0.2 and Saturday's restore drill runs `forgejo doctor` against it, with no second literal to chase.
supernaut kommenterad 2026-08-19 07:55:45 +00:00
Upphovsperson
Ägare
Kopiera länk

Applied to bitborg-prod, 2026-08-19. Patched.

The running container, not the tag:

Before After
podman inspect forgejo ImageName codeberg.org/forgejo/forgejo:16.0.1-rootless codeberg.org/forgejo/forgejo:16.0.2-rootless
gitea --version 16.0.1+gitea-1.22.0 16.0.2+gitea-1.22.0
systemctl --user is-active forgejo active

Checked this way deliberately. #63 records that an image-tag bump can no-op without a pull and
restart, so the declared tag is not evidence. The container is.

The apply

ok=355 changed=7 unreachable=0 failed=0, and the ADR 0038 gate passed rather than being skipped.

apply-reconcile.sh predicted 6 and saw 7, and correctly identified the extra as known-blind rather
than drift:

✓ 1 extra, all on the known-blind list (check mode cannot see these):
    registry-mirror : Run the registry mirror once now (populate before Renovate repoints)
        └─ when: not ansible_check_mode

Useful side effect: the mirror was populated with 16.0.2 in the same run, so CI's container smoke will
find it in-instance rather than falling back upstream.

A second --check then returned changed=0, which is the only thing that proves prod matches main.

Health

  • Core units active: postgres, kanidm, forgejo, bitborg-web, caddy.
  • Public endpoints serving: www.bitborg.se/, www.bitborg.se/healthz, git.bitborg.se/,
    auth.bitborg.se/status.
  • bitborg_unit_active = 1 across all five.
  • All blackbox probes at 1.
  • Only Watchdog firing, severity none, as expected.

Note the pull went to codeberg.org/forgejo/forgejo directly: forgejo_image is the upstream ref, so
the apply never depended on the mirror holding 16.0.2. Worth recording because the reverse would have
failed the apply, and it was checked rather than assumed.

Still open, and it is NOT covered by anything above

EXTERNAL_USER_DISABLE_FEATURES = deletion,manage_password applies only to external OIDC accounts.
The local preview's break-glass admin cannot exercise it, and the ADR 0038 gate only compares tags, so
neither the verification nor this apply says anything about it.

Needs one real Kanidm-backed account checked on git.bitborg.se/user/settings/account: no password
section, no self-delete. If 16.0.2 renamed that key or narrowed its accepted values, external users
have silently regained both, and nothing in the stack would report it.

Follow-up

#446 (no security release can bypass the weekly schedule) is no longer theoretical. This incident is
its evidence: 17 days lost to #430, then the weekly window on top. Worth promoting out of triage.

## Applied to bitborg-prod, 2026-08-19. Patched. The **running container**, not the tag: | | Before | After | | --- | --- | --- | | `podman inspect forgejo` ImageName | `codeberg.org/forgejo/forgejo:16.0.1-rootless` | `codeberg.org/forgejo/forgejo:16.0.2-rootless` | | `gitea --version` | `16.0.1+gitea-1.22.0` | `16.0.2+gitea-1.22.0` | | `systemctl --user is-active forgejo` | | `active` | Checked this way deliberately. #63 records that an image-tag bump can no-op without a pull and restart, so the declared tag is not evidence. The container is. ## The apply `ok=355 changed=7 unreachable=0 failed=0`, and the ADR 0038 gate passed rather than being skipped. `apply-reconcile.sh` predicted 6 and saw 7, and correctly identified the extra as known-blind rather than drift: ``` ✓ 1 extra, all on the known-blind list (check mode cannot see these): registry-mirror : Run the registry mirror once now (populate before Renovate repoints) └─ when: not ansible_check_mode ``` Useful side effect: the mirror was populated with 16.0.2 in the same run, so CI's container smoke will find it in-instance rather than falling back upstream. A second `--check` then returned `changed=0`, which is the only thing that proves prod matches main. ## Health - Core units active: postgres, kanidm, forgejo, bitborg-web, caddy. - Public endpoints serving: `www.bitborg.se/`, `www.bitborg.se/healthz`, `git.bitborg.se/`, `auth.bitborg.se/status`. - `bitborg_unit_active` = 1 across all five. - All blackbox probes at 1. - Only `Watchdog` firing, severity `none`, as expected. Note the pull went to `codeberg.org/forgejo/forgejo` directly: `forgejo_image` is the upstream ref, so the apply never depended on the mirror holding 16.0.2. Worth recording because the reverse would have failed the apply, and it was checked rather than assumed. ## Still open, and it is NOT covered by anything above `EXTERNAL_USER_DISABLE_FEATURES = deletion,manage_password` applies only to **external OIDC** accounts. The local preview's break-glass admin cannot exercise it, and the ADR 0038 gate only compares tags, so neither the verification nor this apply says anything about it. **Needs one real Kanidm-backed account checked on `git.bitborg.se/user/settings/account`: no password section, no self-delete.** If 16.0.2 renamed that key or narrowed its accepted values, external users have silently regained both, and nothing in the stack would report it. ## Follow-up #446 (no security release can bypass the weekly schedule) is no longer theoretical. This incident is its evidence: 17 days lost to #430, then the weekly window on top. Worth promoting out of triage.
Logga in för att delta i denna konversation.
Ingen gren/etikett specificerad
Grenar Taggar
main
fix/log-retention-and-erasure
feat/billing-reconcile-trigger
chore/renovate-script-pins
Inga resultat hittades.
Etiketter
Rensa etiketter
  
area/backups
Backup & restore

  
area/ci
Actions / runners

  
area/control-panel
Logged-in self-service area

  
area/identity
Kanidm / OIDC / entitlements

  
area/infra
Infrastructure (OpenTofu/Ansible/host)

  
area/observability
Monitoring / logs / analytics

  
area/payments
Billing / payment providers

  
area/security
Security & compliance (incl. ISO 27001)

  
area/storage
Volumes / data / LFS

  
area/web
Public web portal (gitborg-web)

  
blocked
Evaluated and specified, but waiting on an external dependency

  
needs-info
Waiting on the reporter for more information

  
needs-triage
Not yet evaluated

  
ready-for-implementation
Fully specified, ready to be picked up

  
type
bug
 Something is broken

  
type
chore
 Maintenance / housekeeping

  
type
docs
 Documentation

  
type
epic
 High-level roadmap item

  
type
feature
 New capability

  
type
task
 Concrete unit of work

  
wontfix
Will not be actioned

Inga etiketter
area/backups
area/ci
area/control-panel
area/identity
area/infra
area/observability
area/payments
area/security
area/storage
area/web
blocked
needs-info
needs-triage
ready-for-implementation
type
bug
type
chore
type
docs
type
epic
type
feature
type
task
wontfix
Milstolpe
Rensa milstolpe
Inga objekt
Ingen milstolpe
Projekt
Rensa projekt
Inga objekt
Inget projekt
Bitborg Infrastructure
Tilldelade
Rensa tilldelade
Inga tilldelade
1 deltagare
Notiser
Förfallodatum
Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens
bitborg/bitborg-infra#447
Ingen beskrivning angiven.