fix(runner-controller): naive OpenStack timestamps disable the volume sweep (#133) #135

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/runner-controller-age-naive-tz in i main

2026-07-19 08:44:12 +00:00

supernaut kommenterad

2026-07-19 08:40:39 +00:00

Ägare

Critical follow-up to #134 — without this the orphan-volume sweep is inert.

Verifying #134 in prod after apply: the new metrics are live and the controller is healthy, but orphan_volumes_swept stayed 0 with 11 detached orphans present. Root cause: OpenStack returns created_at without a timezone (2026-07-19T08:07:59.000000). datetime.fromisoformat parses that as a naive datetime; subtracting it from an aware now() raises TypeError, which _iso_age_seconds's except swallowed into 0.0. So every volume looked 0s old → age <= max_age was always true → the sweep never deleted anything. The same bug sat in _server_age_seconds, silently disabling the reap_max_age server backstop.

Fix: stamp a missing tzinfo as UTC (OpenStack timestamps are UTC), and route _server_age_seconds through the same helper.

Verified by exec'ing the actual function from source against real formats:

'2026-07-19T08:07:59.000000'    -> 1917s   (was 0s — the bug)
'2026-07-19T08:07:59.000000Z'   -> 1917s
'2026-07-19T08:07:59+00:00'     -> 1917s

py_compile clean. Needs the runner-controller image rebuilt + site.yml apply, same as #134. Once live, the sweep will clear the 11 leftover orphans as they cross the 1h age threshold.

**Critical follow-up to #134 — without this the orphan-volume sweep is inert.** Verifying #134 in prod after apply: the new metrics are live and the controller is healthy, but `orphan_volumes_swept` stayed 0 with 11 detached orphans present. Root cause: OpenStack returns `created_at` **without a timezone** (`2026-07-19T08:07:59.000000`). `datetime.fromisoformat` parses that as a **naive** datetime; subtracting it from an aware `now()` raises `TypeError`, which `_iso_age_seconds`'s `except` swallowed into `0.0`. So every volume looked 0s old → `age <= max_age` was always true → the sweep never deleted anything. The same bug sat in `_server_age_seconds`, silently disabling the `reap_max_age` server backstop. Fix: stamp a missing `tzinfo` as UTC (OpenStack timestamps are UTC), and route `_server_age_seconds` through the same helper. Verified by exec'ing the actual function from source against real formats: ``` '2026-07-19T08:07:59.000000' -> 1917s (was 0s — the bug) '2026-07-19T08:07:59.000000Z' -> 1917s '2026-07-19T08:07:59+00:00' -> 1917s ``` `py_compile` clean. Needs the runner-controller image rebuilt + `site.yml` apply, same as #134. Once live, the sweep will clear the 11 leftover orphans as they cross the 1h age threshold.

supernaut lade till 1 incheckning

2026-07-19 08:40:39 +00:00

fix(runner-controller): treat naive OpenStack timestamps as UTC in age calc

ci / ci (pull_request) Successful in 3m0s

Detaljer

9e38f9f3dc

OpenStack returns created_at without a timezone (e.g. 2026-07-19T08:07:59.000000).
fromisoformat parses that as a naive datetime, and subtracting it from an aware
now() raises TypeError — which the except swallowed to 0.0. Every volume/server
then looked 0s old, so the #134 orphan-volume sweep (and the reap_max_age server
backstop) never fired. Stamp a missing tzinfo as UTC. Verified: the naive format
now yields a correct age instead of 0.

supernaut refererade till denna ändringsförfrågan

2026-07-19 08:41:14 +00:00

Ephemeral runners boot-loop to OpenStack ERROR → CI down, silent (no alert), burning volumes #133

supernaut sammanfogade incheckning b3831c4a1f till main

2026-07-19 08:44:12 +00:00

supernaut tog bort grenen fix/runner-controller-age-naive-tz

2026-07-19 08:44:12 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-19 08:44:14 +00:00

fix(runner-controller): naive OpenStack timestamps disable the volume sweep (#133) (#135)

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-19 08:57:24 +00:00