docs(adr-0019): amend for readiness-gated start and automatic rollback #102

Sammanfogat

supernaut sammanfogade 1 incheckning från work/62 in i main

2026-09-21 08:22:37 +00:00

supernaut kommenterad

2026-09-20 22:21:44 +00:00

Ägare

ADR 0019 records the deploy mechanism: CI builds and pushes an image, the services host pulls it via podman auto-update, CI never touches production. Still true. The mechanism has since gained two properties the ADR did not describe, and both change what a deploy means.

The amendment records:

The start is readiness-gated. The web Quadlet unit sets Notify=healthy with a healthcheck against /healthz, which checks the Node process is up and Postgres answers select 1. A start that never becomes ready fails instead of silently serving a broken container.
A failed start rolls back automatically via podman auto-update.
Three things it does not mean: podman auto-update exits 0 whether it deployed or rolled back, so a successful run can mean nothing shipped; a rollback reverts the image but not a migration; web start now depends on Postgres answering, and fails closed during a Postgres outage.

Two Consequences bullets updated to point at it.

Verified against source, not against the issue body: roles/web/templates/bitborg-web.container.j2 for Notify=healthy and the healthcheck, bitborg-web/src/lib/health.ts and src/pages/healthz.ts for what /healthz checks.

The issue's caveat is stale. It said the rollback was verified on podman 5.8.3 but not rehearsed on production's 5.4.2. The runbook's recorded-results table now carries both rows, dated 2026-08-01, and names the 5.4.2 row the load-bearing one. The amendment describes the rollback as rehearsed on a throwaway VM matching production's version, never on production itself.

The post-deploy probe and the Grafana deploy annotation are observability, not mechanism, and are left out.

Closes #62

ADR 0019 records the deploy mechanism: CI builds and pushes an image, the services host pulls it via `podman auto-update`, CI never touches production. Still true. The mechanism has since gained two properties the ADR did not describe, and both change what a deploy means. The amendment records: - The start is readiness-gated. The web Quadlet unit sets `Notify=healthy` with a healthcheck against `/healthz`, which checks the Node process is up and Postgres answers `select 1`. A start that never becomes ready fails instead of silently serving a broken container. - A failed start rolls back automatically via `podman auto-update`. - Three things it does not mean: `podman auto-update` exits 0 whether it deployed or rolled back, so a successful run can mean nothing shipped; a rollback reverts the image but not a migration; web start now depends on Postgres answering, and fails closed during a Postgres outage. Two Consequences bullets updated to point at it. Verified against source, not against the issue body: `roles/web/templates/bitborg-web.container.j2` for `Notify=healthy` and the healthcheck, `bitborg-web/src/lib/health.ts` and `src/pages/healthz.ts` for what `/healthz` checks. The issue's caveat is stale. It said the rollback was verified on podman 5.8.3 but not rehearsed on production's 5.4.2. The runbook's recorded-results table now carries both rows, dated 2026-08-01, and names the 5.4.2 row the load-bearing one. The amendment describes the rollback as rehearsed on a throwaway VM matching production's version, never on production itself. The post-deploy probe and the Grafana deploy annotation are observability, not mechanism, and are left out. Closes #62

supernaut lade till 1 incheckning

2026-09-20 22:21:45 +00:00