registry: no retention policy — CI layer cache fills the [storage] volume at ~0.6 GB/day #297

Nytt ärende

Stängd

öppnade 2026-08-01 09:01:53 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-01 09:01:53 +00:00

Ägare

/srv/gitborg-lfs (Forgejo's [storage] — LFS, packages, attachments, avatars) reached 79% with
4 GB free on 2026-08-01, roughly a week from full, and predict_linear over three days projected it
going negative inside that week. It was grown 20→60 GB as an emergency measure (now 26%), but that
bought runway, it did not fix the leak.

Cause

.forgejo/workflows/deploy.yml in bitborg-web pushes three things on every merge to main: latest,
an immutable sha-XXXX tag, and a registry layer-cache ref via --cache-to. The Forgejo package
registry lives on this volume and nothing prunes it. ARTIFACT_RETENTION_DAYS=30 covers Actions
artifacts, not registry packages.

Measured on 2026-08-01: bitborg-web had 6 versions, but bitborg-web/cache had 44. So the
dominant consumer is the layer cache, not the image tags — worth knowing, because the obvious guess is
wrong. Drift is roughly 0.6 GB/day at the current merge rate.

What to do

A retention policy for registry packages: keep the newest N sha- versions, and collapse or expire
the cache ref. The cache package is safe to delete wholesale at any time — it is rebuilt on the
next push — so it is the easy half.
Do it on a timer, the same shape as the other maintenance units, and export a textfile metric so it
is visible.
Alert on the trend, not just the level: predict_linear on this mountpoint would have caught this
weeks earlier than the 80% threshold did.

Related: #178 covers Actions artifact cleanup, which is a different store and does not address this.

`/srv/gitborg-lfs` (Forgejo's `[storage]` — LFS, packages, attachments, avatars) reached **79% with 4 GB free on 2026-08-01**, roughly a week from full, and `predict_linear` over three days projected it going negative inside that week. It was grown 20→60 GB as an emergency measure (now 26%), but **that bought runway, it did not fix the leak.** ## Cause `.forgejo/workflows/deploy.yml` in bitborg-web pushes three things on every merge to `main`: `latest`, an immutable `sha-XXXX` tag, and a registry layer-cache ref via `--cache-to`. The Forgejo package registry lives on this volume and **nothing prunes it**. `ARTIFACT_RETENTION_DAYS=30` covers Actions artifacts, not registry packages. Measured on 2026-08-01: `bitborg-web` had 6 versions, but **`bitborg-web/cache` had 44**. So the dominant consumer is the layer cache, not the image tags — worth knowing, because the obvious guess is wrong. Drift is roughly 0.6 GB/day at the current merge rate. ## What to do - A retention policy for registry packages: keep the newest N `sha-` versions, and collapse or expire the `cache` ref. The cache package is safe to delete wholesale at any time — it is rebuilt on the next push — so it is the easy half. - Do it on a timer, the same shape as the other maintenance units, and export a textfile metric so it is visible. - Alert on the trend, not just the level: `predict_linear` on this mountpoint would have caught this weeks earlier than the 80% threshold did. Related: #178 covers Actions artifact cleanup, which is a different store and does not address this.

supernaut lade till etiketterna

area/ci

area/storage

ready-for-implementation

type

task

2026-08-01 09:03:50 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-01 09:04:02 +00:00

supernaut refererade till detta ärende från en incheckning

2026-08-01 14:34:54 +00:00

feat(registry): retention policy for the package registry (#297)