monitoring: nothing reconciles away UI-created Grafana dashboards #299

Nytt ärende

Öppen

öppnade 2026-08-01 09:02:26 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-01 09:02:26 +00:00

Ägare

"Dashboards are only defined in code" is true of the current state but not enforced, so it drifts
back silently.

The Grafana file provider sets editable: false and allowUiUpdates: false, which stops edits to
provisioned dashboards. disableDeletion: false only removes a dashboard whose JSON file was
deleted. Neither touches a dashboard the provider never created.

Found 2026-08-01: bitborg CI runners — health existed twice — the provisioned bitborg-runners
and an unprovisioned UI copy (aa433a50-c056-4f2f-ac0e-c329d3f68b3f) created 14 hours earlier by
importing the JSON by hand. Identical title, identical 12 panels. It had survived every apply since
2026-07-20, and two same-titled dashboards make deep links and on-call navigation ambiguous. The
duplicate has been deleted, so the folder is clean today.

Worth recording: an audit of all six provisioned dashboards compared every panel's id, type,
title, unit and expr against the repo JSON and found them byte-identical. The version > 1
values are provisioning reloads, not UI saves. So the provisioning half works well; only the
reconcile-away half is missing.

What to do

Add a converge step to the monitoring role that lists dashboards in the gitborg folder
(/api/search?type=dash-db&folderUIDs=…) and deletes any uid not matching a
bitborg-*.json file in the role's files/. Guard it so it cannot run against an empty file list.

Secondary: schemaVersion is 39 on five files and 42 on bitborg-runners.json, against Grafana
13.0.2. Normalising stops re-provisioning bumping version on schema migration.

"Dashboards are only defined in code" is true of the current state but **not enforced**, so it drifts back silently. The Grafana file provider sets `editable: false` and `allowUiUpdates: false`, which stops edits to *provisioned* dashboards. `disableDeletion: false` only removes a dashboard whose JSON file was deleted. Neither touches a dashboard the provider never created. Found 2026-08-01: `bitborg CI runners — health` existed **twice** — the provisioned `bitborg-runners` and an unprovisioned UI copy (`aa433a50-c056-4f2f-ac0e-c329d3f68b3f`) created 14 hours earlier by importing the JSON by hand. Identical title, identical 12 panels. It had survived every apply since 2026-07-20, and two same-titled dashboards make deep links and on-call navigation ambiguous. The duplicate has been deleted, so the folder is clean today. Worth recording: an audit of all six provisioned dashboards compared every panel's `id`, `type`, `title`, `unit` and `expr` against the repo JSON and found them **byte-identical**. The `version > 1` values are provisioning reloads, not UI saves. So the provisioning half works well; only the reconcile-away half is missing. ## What to do Add a converge step to the `monitoring` role that lists dashboards in the `gitborg` folder (`/api/search?type=dash-db&folderUIDs=…`) and deletes any uid not matching a `bitborg-*.json` file in the role's `files/`. Guard it so it cannot run against an empty file list. Secondary: `schemaVersion` is 39 on five files and 42 on `bitborg-runners.json`, against Grafana 13.0.2. Normalising stops re-provisioning bumping `version` on schema migration.

supernaut lade till etiketterna

area/observability

ready-for-implementation

type

task

2026-08-01 09:03:51 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-08-01 09:04:05 +00:00

Logga in för att delta i denna konversation.