bitborg/bitborg-infra

Bevaka

Avgrena 0

renovate: nightly run OOM-killed at 1.5g — bitborg-web + tenant repos silently skipped #87

Nytt ärende

Stängd

öppnade 2026-07-17 16:41:54 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-07-17 16:41:54 +00:00

Ägare

Found while investigating why bitborg-web stopped getting Renovate PRs.

Root cause: the nightly run is OOM-killed (exit 137) at its own renovate_memory_max: 1.5g cgroup cap, every night since 2026-07-11, exactly while processing bitborg-web (repo 4/5 in autodiscovery order). docs/infra/internal (before it) get PRs; bitborg-web and the tenant repo kofish/hitta_ami (after it) have been silently skipped for a week — this also breaks the tenant-Renovate beta. gitborg_renovate_peak_rss_bytes pins at exactly 1.5 GiB on every failed run; the Loki log shows Killed + exited with status 137 (repos 3) nightly. The bitborg-web Dependency Dashboard body is stale (last successful run predates PR #38 dropping the weekly schedule — red herring).

Fix: raise renovate_memory_max to 3g + add the missing RenovateRunFailed alert on gitborg_renovate_last_run_status (metric existed for exactly this; nothing watched it for 7 days).

Found while investigating why bitborg-web stopped getting Renovate PRs. **Root cause:** the nightly run is OOM-killed (exit 137) at its own `renovate_memory_max: 1.5g` cgroup cap, every night since 2026-07-11, exactly while processing bitborg-web (repo 4/5 in autodiscovery order). docs/infra/internal (before it) get PRs; **bitborg-web and the tenant repo kofish/hitta_ami (after it) have been silently skipped for a week** — this also breaks the tenant-Renovate beta. `gitborg_renovate_peak_rss_bytes` pins at exactly 1.5 GiB on every failed run; the Loki log shows `Killed` + `exited with status 137 (repos 3)` nightly. The bitborg-web Dependency Dashboard body is stale (last successful run predates PR #38 dropping the weekly schedule — red herring). **Fix:** raise `renovate_memory_max` to 3g + add the missing `RenovateRunFailed` alert on `gitborg_renovate_last_run_status` (metric existed for exactly this; nothing watched it for 7 days).

supernaut lade till etiketterna

area/ci

area/observability

ready-for-implementation

type

bug

2026-07-17 16:42:10 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-17 16:42:14 +00:00

fix(renovate): raise memory cap to 3g + alert on failed runs (#87)

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-07-17 16:42:14 +00:00

fix(renovate): raise memory cap to 3g + alert on failed runs (#87) #88

supernaut lade till detta till projektet Bitborg Infrastructure

2026-07-17 16:42:36 +00:00

supernaut stängde ärendet

2026-07-17 20:29:32 +00:00

supernaut refererade till detta ärende från en incheckning

2026-07-17 20:29:33 +00:00

fix(renovate): raise memory cap to 3g + alert on failed runs (#87) (#88)

supernaut refererade till detta ärende från en incheckning

2026-08-03 09:41:33 +00:00

fix(renovate): raise memory cap to 3g + alert on failed runs (#87) (#88)

supernaut ändrade titeln från ~~renovate: nightly run OOM-killed at 1.5g — gitborg-web + tenant repos silently skipped~~ till renovate: nightly run OOM-killed at 1.5g — bitborg-web + tenant repos silently skipped

2026-08-03 09:58:45 +00:00

Logga in för att delta i denna konversation.

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra#87

Ingen beskrivning angiven.

Rader
Kolumner