#221 - feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226) - bitborg/bitborg-infra

supernaut kommenterad

2026-07-26 21:14:05 +00:00

Ägare

⚠️ The ADR 0035 cutover flip — do not merge/apply until you're ready to cut over.

Flips the reconciler from bash-applies / container-dry-run to container-applies, bash retired:

reconciler_next_apply: false → true — the bitborg-auth-reconciler container becomes the applier.
reconciler_apply: true → false — the bash reconciler stops applying (also satisfies the double-reconcile guard the role asserts).
reconciler_enabled: true → false — disables the bash gitborg-reconciler.timer entirely.

Prerequisites (met on main)

Dry-run parallel-run validated: 6+ consecutive cycles where the container's proposed projections matched the bash reconciler's applied ones, per user (#226).
Metric-write fix (keep-id, #218) and backup-drill fix (#220) merged to main, so applying this branch picks them up too.

Apply (guided, when ready)

Merge + pull. 2. (Recommended) run one more dry-run cycle post-#218 and confirm reconciler-next/reconciler.prom writes cleanly + the diff still matches. 3. site.yml --tags reconciler --check --diff (review) → apply.
Verify: container applies (RECONCILE_APPLY=true, real actions in journal), bash gitborg-reconciler.timer inactive, a spot-checked user's entitlements unchanged, no ReconcilerStale (the container now feeds that metric).

Backout

Revert this commit (re-enable bash + container back to dry-run) + apply.

Follow-up (after burn-in)

Remove the bash gitborg-reconciler.sh.j2 + its templates/tasks in a cleanup PR.

Refs #226, ADR 0035.

⚠️ **The ADR 0035 cutover flip — do not merge/apply until you're ready to cut over.** Flips the reconciler from *bash-applies / container-dry-run* to **container-applies, bash retired**: - `reconciler_next_apply: false → true` — the `bitborg-auth-reconciler` container becomes the applier. - `reconciler_apply: true → false` — the bash reconciler stops applying (also satisfies the double-reconcile guard the role asserts). - `reconciler_enabled: true → false` — disables the bash `gitborg-reconciler.timer` entirely. ## Prerequisites (met on main) - Dry-run parallel-run validated: 6+ consecutive cycles where the container's proposed projections matched the bash reconciler's applied ones, per user (#226). - Metric-write fix (keep-id, #218) and backup-drill fix (#220) merged to main, so applying this branch picks them up too. ## Apply (guided, when ready) 1. Merge + pull. 2. (Recommended) run one more dry-run cycle post-#218 and confirm `reconciler-next/reconciler.prom` writes cleanly + the diff still matches. 3. `site.yml --tags reconciler --check --diff` (review) → apply. 4. Verify: container applies (`RECONCILE_APPLY=true`, real actions in journal), bash `gitborg-reconciler.timer` inactive, a spot-checked user's entitlements unchanged, no `ReconcilerStale` (the container now feeds that metric). ## Backout Revert this commit (re-enable bash + container back to dry-run) + apply. ## Follow-up (after burn-in) Remove the bash `gitborg-reconciler.sh.j2` + its templates/tasks in a cleanup PR. Refs #226, ADR 0035.

supernaut refererade till denna ändringsförfrågan

2026-07-27 12:17:02 +00:00

Extract the Kanidm→Forgejo entitlement reconciler into its own repo (bitborg-auth-reconciler, ADR 0035) #226

supernaut ändrade titeln från ~~WIP: feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#202)~~ till WIP: feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226)

2026-07-27 12:18:24 +00:00

supernaut tvångsskickade feat/202-reconciler-flip-to-apply från 04ae1199e9

ci / ci (pull_request) Successful in 1m28s

Detaljer

till 78169b67e9

ci / ci (pull_request) Successful in 1m27s

Detaljer

2026-07-27 12:55:09 +00:00

Jämför

supernaut kommenterad

2026-07-27 12:57:31 +00:00

Upphovsperson

Ägare

✅ Cutover applied + verified on prod (2026-07-27)

Rebased onto main (picks up #223/#225) and applied site.yml --tags reconciler. Two fixes were folded in beyond the original 3-boolean flip — without them the cutover would have broken monitoring:

Container takes over the scraped reconciler.prom. reconciler_next_textfile_dir now points at the scraped node_textfile_dir (was a dedicated, deliberately-unscraped dir for the parallel run). The container writes the same basename + metric names at 0644, so the ReconcilerStale/ReconcilerFailed/ReconcilerActionsEnforcementDegraded deadman feed keeps ticking. Safe only because the bash writer is retired in the same apply (no duplicate-metric collision). Without this, the scraped file would freeze and ReconcilerStale would fire ~30 min post-cutover while the container's real metrics went unscraped.
Dir-mode clamp fixed (caught by --check --diff). The reconciler-next dir task hardcoded mode: 0700; repointed at the shared scraped dir it would have clamped node_textfile_dir 0755 → 0700, locking node_exporter out of every bitborg_* textfile metric. Split the task: config dir stays 0700, textfile dir is 0755 (matches the monitoring-agent/runner-controller canonical — idempotent, no flap).

Runbook cutover step updated to include the repoint + backout.

Verification

Bash gitborg-reconciler.timer → inactive + disabled (retired).
Container gitborg-reconciler-next.timer → active + enabled; on-demand run exited 0, log ends reconcile complete (APPLY=true), sane per-user projections, legacy LFS groups purged, no errors.
Scraped reconciler.prom → container-owned, fresh, 0644, status=0, degraded=0.
Prometheus → fresh single series (no collision), staleness ~1.1 min ≪ 30 → no reconciler alert firing.
Health gates green (incl. "textfile metrics world-readable"); re-apply idempotent (changed=0).

Follow-up (after burn-in): remove the bash gitborg-reconciler.sh.j2 + templates/tasks, and drop the now-stale "DRY-RUN parallel run" labels from the container/timer unit descriptions (it's the sole reconciler now, not "next"). Tracked on #226.

Refs #226, ADR 0035.

## ✅ Cutover applied + verified on prod (2026-07-27) Rebased onto `main` (picks up #223/#225) and applied `site.yml --tags reconciler`. **Two fixes were folded in beyond the original 3-boolean flip** — without them the cutover would have broken monitoring: 1. **Container takes over the scraped `reconciler.prom`.** `reconciler_next_textfile_dir` now points at the scraped `node_textfile_dir` (was a dedicated, deliberately-unscraped dir for the parallel run). The container writes the same basename + metric names at `0644`, so the `ReconcilerStale`/`ReconcilerFailed`/`ReconcilerActionsEnforcementDegraded` deadman feed keeps ticking. Safe only because the bash writer is retired in the same apply (no duplicate-metric collision). **Without this, the scraped file would freeze and `ReconcilerStale` would fire ~30 min post-cutover** while the container's real metrics went unscraped. 2. **Dir-mode clamp fixed** (caught by `--check --diff`). The reconciler-next dir task hardcoded `mode: 0700`; repointed at the shared scraped dir it would have clamped `node_textfile_dir` `0755 → 0700`, locking node_exporter out of **every** `bitborg_*` textfile metric. Split the task: config dir stays `0700`, textfile dir is `0755` (matches the `monitoring-agent`/`runner-controller` canonical — idempotent, no flap). Runbook cutover step updated to include the repoint + backout. ### Verification - Bash `gitborg-reconciler.timer` → **inactive + disabled** (retired). - Container `gitborg-reconciler-next.timer` → **active + enabled**; on-demand run exited `0`, log ends `reconcile complete (APPLY=true)`, sane per-user projections, legacy LFS groups purged, no errors. - Scraped `reconciler.prom` → container-owned, fresh, `0644`, `status=0`, `degraded=0`. - Prometheus → fresh **single** series (no collision), staleness ~1.1 min ≪ 30 → no reconciler alert firing. - Health gates green (incl. "textfile metrics world-readable"); re-apply idempotent (`changed=0`). **Follow-up (after burn-in):** remove the bash `gitborg-reconciler.sh.j2` + templates/tasks, and drop the now-stale "DRY-RUN parallel run" labels from the container/timer unit descriptions (it's the sole reconciler now, not "next"). Tracked on #226. Refs #226, ADR 0035.

supernaut ändrade titeln från ~~WIP: feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226)~~ till feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226)

2026-07-27 12:58:19 +00:00

supernaut sammanfogade incheckning 5be67bb72c till main

2026-07-27 12:58:47 +00:00

supernaut tog bort grenen feat/202-reconciler-flip-to-apply

2026-07-27 12:58:48 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-27 12:58:48 +00:00

feat(reconciler): flip ADR 0035 cutover — container becomes sole applier (#226) (#221)

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-27 19:05:28 +00:00

refactor(reconciler): retire the bash reconciler after ADR 0035 cutover (#226)

supernaut refererade till denna ändringsförfrågan