fix(runner-controller): sweep skips snapshot-backed volumes, no retry-spam (#133) #136

Sammanfogat

supernaut sammanfogade 1 incheckning från fix/runner-controller-sweep-skip-snapshots in i main

2026-07-19 09:26:16 +00:00

supernaut kommenterad

2026-07-19 08:57:46 +00:00

Ägare

Third and final refinement to the #133 sweep (follows #134 sweep+alerts, #135 tz fix).

What post-#135 verification found: the sweep now ages orphans correctly (the tz fix works) and deletes plain runner leaks — but it hammered two unnamed detached volumes every 10s with:

400 Invalid volume: … must not … have snapshots or belong to a group …

Those two aren't ephemeral leaks: a runner boot volume never has a snapshot. They carry snapshots — the gitborg-runner image's source snapshot and the predrill-* 2026-07-09 safety snapshots — i.e. image/backup infrastructure. OpenStack correctly refused; the blank-name heuristic was just too broad.

Fix:

Skip any volume that has snapshots (one snapshots() list call per cycle, not per-volume; falls back gracefully if that call fails).
Skip-list (in-memory) any volume whose delete is refused for any reason (group membership, attach race), so a guaranteed-failing delete isn't retried — or re-logged — every cycle.

Validated: py_compile + ansible-lint (production) clean, and a stub-connection unit test of the actual function asserts it deletes a plain old orphan, skips+skip-lists a snapshot-backed volume (no re-attempt on the next cycle), and never touches named/in-use/too-young volumes.

Needs image rebuild + site.yml apply. After it, the 10 genuine runner-leak orphans (currently aging past the 1h threshold) sweep cleanly and the two infra volumes are left alone silently.

Third and final refinement to the #133 sweep (follows #134 sweep+alerts, #135 tz fix). **What post-#135 verification found:** the sweep now ages orphans correctly (the tz fix works) and deletes plain runner leaks — but it hammered **two unnamed detached volumes every 10s** with: ``` 400 Invalid volume: … must not … have snapshots or belong to a group … ``` Those two aren't ephemeral leaks: a runner boot volume never has a snapshot. They carry snapshots — the `gitborg-runner` image's source snapshot and the `predrill-*` 2026-07-09 safety snapshots — i.e. **image/backup infrastructure**. OpenStack correctly refused; the blank-name heuristic was just too broad. **Fix:** - Skip any volume that has snapshots (one `snapshots()` list call per cycle, not per-volume; falls back gracefully if that call fails). - Skip-list (in-memory) any volume whose delete is refused for any reason (group membership, attach race), so a guaranteed-failing delete isn't retried — or re-logged — every cycle. **Validated:** `py_compile` + `ansible-lint` (production) clean, and a stub-connection unit test of the actual function asserts it deletes a plain old orphan, skips+skip-lists a snapshot-backed volume (no re-attempt on the next cycle), and never touches named/in-use/too-young volumes. Needs image rebuild + `site.yml` apply. After it, the 10 genuine runner-leak orphans (currently aging past the 1h threshold) sweep cleanly and the two infra volumes are left alone silently.

supernaut lade till 1 incheckning

2026-07-19 08:57:46 +00:00

fix(runner-controller): sweep skips volumes with snapshots + won't retry stuck deletes

ci / ci (pull_request) Successful in 2m38s

Detaljer

8927343286

Post-deploy of #135 the sweep correctly aged out orphans but hammered two unnamed
detached volumes every 10s with a 400 (Cinder refuses to delete a volume that has
snapshots). Those aren't ephemeral leaks — a runner boot volume never has a
snapshot; they're image/backup infra (the gitborg-runner image source snapshot +
pre-drill safety snapshots). Skip any volume that has snapshots, and skip-list any
volume whose delete is refused so a guaranteed-failing delete isn't retried/re-logged
each cycle. Unit-tested the logic with a stub connection.

supernaut refererade till denna ändringsförfrågan

2026-07-19 08:58:46 +00:00

Ephemeral runners boot-loop to OpenStack ERROR → CI down, silent (no alert), burning volumes #133

supernaut refererade till denna ändringsförfrågan

2026-07-19 09:23:26 +00:00

smoke-containers: pull base images from the in-instance registry mirror (CI resilience + sovereignty) #137

supernaut sammanfogade incheckning 8d1ac4ee0a till main

2026-07-19 09:26:16 +00:00

supernaut tog bort grenen fix/runner-controller-sweep-skip-snapshots

2026-07-19 09:26:16 +00:00

supernaut refererade denna ändringsförfrågan från en incheckning

2026-07-19 09:26:16 +00:00