bitborg/bitborg-infra

Bevaka

Avgrena 0

tofu: runner and drill security groups keep Neutron's default allow-all egress #471

Nytt ärende

Stängd

öppnade 2026-09-08 23:06:02 +00:00 av supernaut · 2 kommentarer

supernaut kommenterad

2026-09-08 23:06:02 +00:00

Ägare

opentofu/ephemeral-runners.tofu and opentofu/backup-drill.tofu declare egress rules for 443 and 53 only and describe the groups that way. Neither sets delete_default_rules = true, so Neutron's default allow-all IPv4 and IPv6 egress rules stay attached and the explicit rules restrict nothing. CI runner VMs run job code with unrestricted outbound.

Deleting the defaults is a two-line change but not a no-op:

cloud-init fetches user-data from 169.254.169.254:80 at boot; without an explicit rule the runner never registers.
Jobs that apt-get install reach Debian mirrors over port 80.
NTP is 123/udp.

Plan: add explicit rules for metadata (80/tcp to 169.254.169.254/32) and NTP, decide whether port 80 to the world is allowed for jobs or whether the runner image bakes what jobs need, then set delete_default_rules = true on both groups and boot one runner and one drill VM to prove it. Verify the live rule set first with openstack security group show.

`opentofu/ephemeral-runners.tofu` and `opentofu/backup-drill.tofu` declare egress rules for 443 and 53 only and describe the groups that way. Neither sets `delete_default_rules = true`, so Neutron's default allow-all IPv4 and IPv6 egress rules stay attached and the explicit rules restrict nothing. CI runner VMs run job code with unrestricted outbound. Deleting the defaults is a two-line change but not a no-op: - cloud-init fetches user-data from `169.254.169.254:80` at boot; without an explicit rule the runner never registers. - Jobs that `apt-get install` reach Debian mirrors over port 80. - NTP is 123/udp. Plan: add explicit rules for metadata (80/tcp to `169.254.169.254/32`) and NTP, decide whether port 80 to the world is allowed for jobs or whether the runner image bakes what jobs need, then set `delete_default_rules = true` on both groups and boot one runner and one drill VM to prove it. Verify the live rule set first with `openstack security group show`.

supernaut lade till etiketterna

2026-09-08 23:06:02 +00:00

supernaut lade till detta till projektet Bitborg Infrastructure

2026-09-08 23:06:04 +00:00

supernaut refererade till detta ärende från en incheckning

2026-09-19 12:58:14 +00:00

fix(tofu): drop Neutron's default allow-all egress on the runner and drill groups

supernaut hänvisade till detta ärende från en ändringsförfrågan som kommer att stänga det

2026-09-19 12:58:29 +00:00

fix(tofu): drop Neutron's default allow-all egress on the runner and drill groups #483

supernaut

2026-09-19 13:16:05 +00:00

stängde ärendet
lade till
ready-for-implementation
och tog bort
needs-triage
etiketter

supernaut kommenterad

2026-09-19 20:53:30 +00:00

Upphovsperson

Ägare

Applied and verified on 2026-09-19 (PR #483).

Live rule set after the apply, both groups, with Neutron's default allow-all IPv4 and IPv6 egress gone:

gitborg-prod-ephemeral-runner-sg   443 v4/v6, 53 tcp+udp v4/v6, 80 -> 169.254.169.254/32, 123 v4/v6
gitborg-prod-backup-drill-sg       443 v4/v6, 53 tcp+udp v4, 80 -> 169.254.169.254/32, 123 v4/v6,
                                   22 + 2222 from 192.168.42.0/24

Runner proof: the CI run for PR #485 booted ephemeral-runner-0a06c154 against the new group and completed green, so cloud-init, runner registration and the job itself all work without the default egress. The drill group's proof folds into the next scheduled drill.

One side effect worth recording. delete_default_rules forces replacement of the group, and the old drill group also carried ingress on 22 and 2222 from 155.4.69.98/32. Those rules were never in the tofu state (tofu state list shows nine rules plus the group, none of them a 155 prefix) and local.backup_drill_ssh_cidr falls back to var.subnet_cidr, so they were added out-of-band at some point and the replacement swept them. Drill VMs get no floating IP, so a public-source ingress rule had nothing to match; the deletion looks like cleanup rather than a regression. If direct admin SSH to a drill VM is ever wanted, declare it in tofu rather than adding it by hand.

Applied and verified on 2026-09-19 (PR #483). Live rule set after the apply, both groups, with Neutron's default allow-all IPv4 and IPv6 egress gone: ``` gitborg-prod-ephemeral-runner-sg 443 v4/v6, 53 tcp+udp v4/v6, 80 -> 169.254.169.254/32, 123 v4/v6 gitborg-prod-backup-drill-sg 443 v4/v6, 53 tcp+udp v4, 80 -> 169.254.169.254/32, 123 v4/v6, 22 + 2222 from 192.168.42.0/24 ``` Runner proof: the CI run for PR #485 booted `ephemeral-runner-0a06c154` against the new group and completed green, so cloud-init, runner registration and the job itself all work without the default egress. The drill group's proof folds into the next scheduled drill. One side effect worth recording. `delete_default_rules` forces replacement of the group, and the old drill group also carried ingress on 22 and 2222 from `155.4.69.98/32`. Those rules were never in the tofu state (`tofu state list` shows nine rules plus the group, none of them a 155 prefix) and `local.backup_drill_ssh_cidr` falls back to `var.subnet_cidr`, so they were added out-of-band at some point and the replacement swept them. Drill VMs get no floating IP, so a public-source ingress rule had nothing to match; the deletion looks like cleanup rather than a regression. If direct admin SSH to a drill VM is ever wanted, declare it in tofu rather than adding it by hand.

supernaut kommenterad

2026-09-20 08:46:23 +00:00

Upphovsperson

Ägare

Drill-side proof, 2026-09-20. A manual bitborg-backup-drill.service run booted a VM under the replaced gitborg-prod-backup-drill-sg and passed:

[drill]   drill VM b014ab70-4269-479c-bf6d-69be5fade83b up at 192.168.42.185
[restore] starting scratch postgres (docker.io/library/postgres:17.11-trixie)
[restore] OK — off-site archive restored and forgejo doctor passed with no errors
[drill]   restore-drill PASSED
[drill]   teardown: deleted drill VM b014ab70-... (boot volume cascades)

So the VM reached the metadata service at boot, resolved DNS, and pulled images over 443 with the default allow-all egress gone. No VM or boot volume leaked.

Both groups are now proven under a real workload: the runner group by the CI run for PR #485, the drill group by this.

Drill-side proof, 2026-09-20. A manual `bitborg-backup-drill.service` run booted a VM under the replaced `gitborg-prod-backup-drill-sg` and passed: ``` [drill] drill VM b014ab70-4269-479c-bf6d-69be5fade83b up at 192.168.42.185 [restore] starting scratch postgres (docker.io/library/postgres:17.11-trixie) [restore] OK — off-site archive restored and forgejo doctor passed with no errors [drill] restore-drill PASSED [drill] teardown: deleted drill VM b014ab70-... (boot volume cascades) ``` So the VM reached the metadata service at boot, resolved DNS, and pulled images over 443 with the default allow-all egress gone. No VM or boot volume leaked. Both groups are now proven under a real workload: the runner group by the CI run for PR #485, the drill group by this.

Logga in för att delta i denna konversation.

Ingen milstolpe

Inget projekt

Inga tilldelade

1 deltagare

Notiser

Förfallodatum

Förfallodatumet är ogiltigt eller utanför gränserna. Använd formatet "åååå-mm-dd".

Inget förfallodatum satt.

Beroenden

Inga beroenden satta

Referens

bitborg/bitborg-infra#471

Ingen beskrivning angiven.

Rader
Kolumner