fix(mail): rotate the Sweego credentials + correct the §5 guidance #385

Sammanfogat

supernaut sammanfogade 2 incheckningar från fix/sweego-credentials-rotated in i main

2026-08-05 19:51:45 +00:00

supernaut kommenterad

2026-08-05 19:49:32 +00:00

Ägare

Rotates the Sweego API key and SMTP credentials, and corrects the §5 guidance that sent this
investigation the wrong way.

Already applied to production (Forgejo, portal and Alertmanager restarted; both hosts reconverge
at changed=0). This PR brings main in line with what is deployed — the vault edit was made
directly on the host-facing tree while portal mail was down.

The failure

Portal mail was returning 401 on every sender, including alerts@mail.gitborg.se, which had
returned 200 hours earlier. Meanwhile:

the key value was unchanged,
it was correctly deployed — the vaulted value and the value inside the running container had
identical sha256 fingerprints,
and the Sweego dashboard showed both sending domains allowed, for both API and SMTP.

Freshly minted credentials fixed it immediately. The old key had been invalidated provider-side
without its value changing and without the domain list reflecting it.

Portal mail is API-only (src/lib/email.ts), so while this was broken new sign-ups received no
setup link and could not complete registration.

What the guidance said, and why it was wrong

Two places stated that the API key was not the problem, must not be re-scoped or re-minted, and
that no vault edit should be expected. The measurement underneath was correct — authorisation is
per-domain, not per-address — but the inference was not: that the key authorised the old domain
did not make the key healthy. Corrected in place rather than deleted, because the reasoning looked
sound and will look sound again.

Why a 401 cannot be diagnosed from a host

Recorded so nobody re-derives it. Sweego returns an identical 401 {"detail":"Unauthorized"} for an
invalid key and for a valid key with an unauthorised sender. Every route that might have
discriminated was probed:

probe	result
`/contacts` `/lists` `/campaigns` `/templates` `/webhooks` `/transactions` `/messages` `/segments` `/emails` `/suppressions` `/bounces` `/domains` `/senders` `/sending-domains` `/identities` `/account` `/me` `/transac/domains`	`404 Route Not Found`
`/channels`	200 — but UNAUTHENTICATED
`/stats`	requires auth, but `401` for any key
`/send` with empty body	`422` for any key — body validated before credential

/channels is the trap: it returns the same 200 body with a garbage key, an empty key, and no
header at all. It nearly read as proof the key was valid, and was only caught because the negative
control was run.

The only decisive test is a freshly minted key. The guidance now says to re-mint early.

Also added

Gate step 0 before the apply, not after. Step 0 can be run against the vaulted value from the
prod host with a throwaway play (uri + no_log on the request, failed_when: false, debug the
status only), proving the credential before anything restarts. That is how this rotation was
verified — both domains returned 200 with transaction ids before the apply, so no restart was
spent on an untested key.

Side effect: §5 is unblocked

email.bitborg.se now returns 200 for the first time, so step 0 passes and steps 1–3 of the
mail-domain move are available. None of them are done here — step 1 remains applied, step 2 stays
reverted, step 3 has never been applied.

Note on the rotation mechanism

forgejo_mailer_passwd_secret is content-addressed:

gitborg-forgejo-mailer-passwd-{{ (vault_forgejo_mailer_passwd | hash('sha1'))[:12] }}

Podman secrets cannot be mutated in place, so rotating the password renames the secret, which
changes the Quadlet unit, which restarts Forgejo onto it. Two of the apply's changed tasks are
consequences of that one root cause, not independent drift.

Rotates the Sweego API key and SMTP credentials, and corrects the §5 guidance that sent this investigation the wrong way. **Already applied to production** (Forgejo, portal and Alertmanager restarted; both hosts reconverge at `changed=0`). This PR brings `main` in line with what is deployed — the vault edit was made directly on the host-facing tree while portal mail was down. ## The failure Portal mail was returning `401` on **every** sender, including `alerts@mail.gitborg.se`, which had returned `200` hours earlier. Meanwhile: - the key value was **unchanged**, - it was **correctly deployed** — the vaulted value and the value inside the running container had identical `sha256` fingerprints, - and the Sweego dashboard showed **both** sending domains allowed, for both API and SMTP. Freshly minted credentials fixed it immediately. The old key had been invalidated provider-side without its value changing and without the domain list reflecting it. Portal mail is API-only (`src/lib/email.ts`), so while this was broken **new sign-ups received no setup link and could not complete registration**. ## What the guidance said, and why it was wrong Two places stated that the API key was *not* the problem, must *not* be re-scoped or re-minted, and that *no vault edit* should be expected. The measurement underneath was correct — authorisation is per-domain, not per-address — but the inference was not: **that the key authorised the old domain did not make the key healthy.** Corrected in place rather than deleted, because the reasoning looked sound and will look sound again. ## Why a 401 cannot be diagnosed from a host Recorded so nobody re-derives it. Sweego returns an identical `401 {"detail":"Unauthorized"}` for an invalid key **and** for a valid key with an unauthorised sender. Every route that might have discriminated was probed: | probe | result | | --- | --- | | `/contacts` `/lists` `/campaigns` `/templates` `/webhooks` `/transactions` `/messages` `/segments` `/emails` `/suppressions` `/bounces` `/domains` `/senders` `/sending-domains` `/identities` `/account` `/me` `/transac/domains` | `404 Route Not Found` | | `/channels` | **200 — but UNAUTHENTICATED** | | `/stats` | requires auth, but `401` for any key | | `/send` with empty body | `422` for any key — body validated *before* credential | `/channels` is the trap: it returns the same 200 body with a garbage key, an empty key, and **no header at all**. It nearly read as proof the key was valid, and was only caught because the negative control was run. **The only decisive test is a freshly minted key.** The guidance now says to re-mint early. ## Also added **Gate step 0 before the apply, not after.** Step 0 can be run against the *vaulted* value from the prod host with a throwaway play (`uri` + `no_log` on the request, `failed_when: false`, debug the status only), proving the credential before anything restarts. That is how this rotation was verified — both domains returned `200` with transaction ids *before* the apply, so no restart was spent on an untested key. ## Side effect: §5 is unblocked `email.bitborg.se` now returns `200` for the first time, so **step 0 passes** and steps 1–3 of the mail-domain move are available. None of them are done here — step 1 remains applied, step 2 stays reverted, step 3 has never been applied. ## Note on the rotation mechanism `forgejo_mailer_passwd_secret` is content-addressed: ``` gitborg-forgejo-mailer-passwd-{{ (vault_forgejo_mailer_passwd | hash('sha1'))[:12] }} ``` Podman secrets cannot be mutated in place, so rotating the password renames the secret, which changes the Quadlet unit, which restarts Forgejo onto it. Two of the apply's changed tasks are consequences of that one root cause, not independent drift.

supernaut lade till 2 incheckningar

2026-08-05 19:49:32 +00:00

fix(mail): rotate the Sweego API key and SMTP credentials 40401fba35

Portal mail had been failing with 401 on EVERY sender, including addresses that
returned 200 the day before. The key value in the vault was intact and correctly
deployed (vault and container fingerprints matched), and the Sweego dashboard
showed both sending domains allowed for both API and SMTP — yet /send refused
everything, and so did /stats, an authenticated route with no sender involved.

Freshly minted credentials fixed it immediately. The old key had been invalidated
provider-side without its value changing and without the domain list reflecting
it, so nothing observable from our side could distinguish it from an
authorisation problem.

Side effect worth having: email.bitborg.se now returns 200 as well, so the §5
step-0 gate passes for the first time and the mail-domain move is unblocked.

The mailer secret name is content-addressed
(gitborg-forgejo-mailer-passwd-<sha1[:12]>), so rotating the password renames the
podman secret, changes the Quadlet unit and restarts Forgejo onto it
automatically.

docs(mail): correct the §5 guidance — re-minting the key is what fixed it

ci / ci (pull_request) Successful in 1m31s

Detaljer

4fe417d698

The previous guidance stated, confidently and in two places, that the API key was
NOT the problem and must not be re-scoped or re-minted, and that no vault edit
should be expected. All three claims were wrong, and following them cost an
evening ruling out the one thing that worked.

The measurement underneath was sound — authorisation is per-domain, not
per-address — but the inference from it was not: that the key authorised the old
domain did not make the key healthy. Hours later every sender 401'd, including one
that had returned 200 that morning, while the key value was unchanged, correctly
deployed (vault and container sha256 fingerprints identical) and the dashboard
still showed both domains allowed for API and SMTP.

Also records why a 401 here cannot be diagnosed from a host, with the full route
survey: Sweego returns the same 401 for an invalid key and for a valid key with an
unauthorised sender; /send validates the body BEFORE the credential so an empty
body is 422 either way; and /channels answers 200 while being entirely
unauthenticated — it returns the same body with a garbage key, an empty key and no
header at all, which is a trap worth naming because it nearly read as proof the
key was good.

Step 0 now passes on both domains, so §5 steps 1-3 are unblocked. Adds the point
that step 0 should gate the apply rather than follow it: it can be run against the
vaulted value before anything restarts, and rotating these credentials restarts
Forgejo, the portal and Alertmanager.

supernaut sammanfogade incheckning 14520d1479 till main

2026-08-05 19:51:45 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-08-05 19:51:46 +00:00

fix(mail): rotate the Sweego credentials + correct the §5 guidance (#385)

supernaut refererade till denna ändringsförfrågan

2026-08-05 20:01:31 +00:00

docs(mail): record that the SMTP path is verified for both domains #386

supernaut refererade till denna ändringsförfrågan

2026-08-05 20:06:07 +00:00

mail: verify email.bitborg.se as a Sweego sending domain before re-landing the move #381

supernaut refererade denna ändringsförfrågan från en incheckning