chore(test): replace Nightwatch with Playwright as the end-to-end framework #174

Nytt ärende

Stängd

öppnade 2026-08-02 16:06:37 +00:00 av supernaut · 0 kommentarer

supernaut kommenterad

2026-08-02 16:06:37 +00:00

Ägare

Three things motivate the swap beyond taste:

nightwatch is declared in dependencies, not devDependencies, and the Containerfile runs
pnpm install --prod in both stages — so the container serving www.gitborg.se ships a WebDriver
client and its closure (selenium-webdriver 18 MB, nightwatch 3.3 MB, axe-core 2.9 MB, and their
transitive deps). None of it has a runtime purpose.
WebKit coverage. Issue #158 (the skip link not moving focus) reproduced in WebKit only, and on
iOS every browser is WebKit. geckodriver cannot reach that engine; Playwright can.
scripts/e2e.mjs is 77 lines of hand-rolled build / spawn / wait-for-ready / teardown that
Playwright's webServer config block replaces with about eight.

Scope

Add @playwright/test and @axe-core/playwright to devDependencies.
Add playwright.config.ts. Keep the existing conventions: port from E2E_PORT, default 4331
(not the dev server's 4321), 127.0.0.1 rather than localhost (on macOS localhost can
resolve to ::1 while the server binds IPv4). Set locale: "sv-SE" and
timezoneId: "Europe/Stockholm" on the default context — the site is Swedish-first, and pinning
them stops a UTC runner and a CEST laptop disagreeing.
Projects: chromium, webkit, and mobile-safari (devices["iPhone 15"], which reuses the
WebKit binary and costs no extra download) on every run; firefox only when
E2E_FULL_MATRIX=true. Firefox is what Nightwatch drove, so it is kept, but no Gecko-only defect
has ever been found here and it is a 108 MB download — it belongs on the push: main run rather
than on every pull request.
Port src/test/e2e/homepage.js to homepage.spec.ts, doubled across sv and en. Use role
locators (getByRole("heading", { level: 1 })) rather than the "h1" CSS selector.
Delete nightwatch.conf.js, scripts/e2e.mjs, src/test/e2e/homepage.js; drop nightwatch
from dependencies and geckodriver from devDependencies.
Remove the "Nightwatch e2e globals" block from eslint.config.mjs and the nightwatch.conf.js
entry from the node-globals block below it. Specs are TypeScript with real imports, so no globals
injection is needed.
Remove the @nightwatch/nightwatch-inspector and geckodriver entries from
pnpm-workspace.yaml's allowBuilds. Do not add a playwright entry — @playwright/test
has no postinstall browser download, which is a supply-chain improvement over geckodriver's.
Migrate tests_output/ → playwright-report/ + test-results/ in .gitignore,
.prettierignore and .stylelintignore (and fix the stale "nightwatch" comment in .gitignore).
Add an explicit exclude for src/test/e2e/** in vitest.config.ts so the two runners can never
collide over a filename.
Scripts: test:e2e → playwright test; add test:e2e:ui and test:e2e:report.
Narrow the pre-push e2e job in lefthook.json to --project=chromium so the hook stays fast.
Document the suite in README.md / CONTRIBUTING.md — neither mentions end-to-end tests today.

Sequencing

Commit in three steps, deletion last, so a reviewer can stop after step 1 with both frameworks
working: (1) add Playwright alongside and port the spec; (2) add the new specs (separate issue);
(3) remove Nightwatch.

Out of scope

Wiring e2e into CI, and baking browsers into the runner image — both are separate issues, and the
image must be baked before CI depends on it.

Acceptance

pnpm test:e2e passes on chromium, webkit and mobile-safari
E2E_FULL_MATRIX=true pnpm test:e2e also passes on firefox
pnpm lint, pnpm check, pnpm test pass
grep -ri nightwatch over the working tree returns nothing but the lockfile's unrelated
vitest peer entries
A --prod install tree no longer contains selenium-webdriver

The end-to-end suite is one Nightwatch spec with three assertions, copied from the Astro guide, driving whatever Firefox happens to be installed on the developer's machine. It has never run in CI — it is enforced only by the `pre-push` hook in `lefthook.json`. Replace it with Playwright. Three things motivate the swap beyond taste: 1. `nightwatch` is declared in `dependencies`, not `devDependencies`, and the Containerfile runs `pnpm install --prod` in both stages — so the container serving www.gitborg.se ships a WebDriver client and its closure (selenium-webdriver 18 MB, nightwatch 3.3 MB, axe-core 2.9 MB, and their transitive deps). None of it has a runtime purpose. 2. WebKit coverage. Issue #158 (the skip link not moving focus) reproduced in WebKit only, and on iOS every browser is WebKit. geckodriver cannot reach that engine; Playwright can. 3. `scripts/e2e.mjs` is 77 lines of hand-rolled build / spawn / wait-for-ready / teardown that Playwright's `webServer` config block replaces with about eight. ### Scope - Add `@playwright/test` and `@axe-core/playwright` to **devDependencies**. - Add `playwright.config.ts`. Keep the existing conventions: port from `E2E_PORT`, default 4331 (not the dev server's 4321), `127.0.0.1` rather than `localhost` (on macOS `localhost` can resolve to ::1 while the server binds IPv4). Set `locale: "sv-SE"` and `timezoneId: "Europe/Stockholm"` on the default context — the site is Swedish-first, and pinning them stops a UTC runner and a CEST laptop disagreeing. - Projects: `chromium`, `webkit`, and `mobile-safari` (`devices["iPhone 15"]`, which reuses the WebKit binary and costs no extra download) on every run; `firefox` only when `E2E_FULL_MATRIX=true`. Firefox is what Nightwatch drove, so it is kept, but no Gecko-only defect has ever been found here and it is a 108 MB download — it belongs on the `push: main` run rather than on every pull request. - Port `src/test/e2e/homepage.js` to `homepage.spec.ts`, doubled across `sv` and `en`. Use role locators (`getByRole("heading", { level: 1 })`) rather than the `"h1"` CSS selector. - Delete `nightwatch.conf.js`, `scripts/e2e.mjs`, `src/test/e2e/homepage.js`; drop `nightwatch` from `dependencies` and `geckodriver` from devDependencies. - Remove the "Nightwatch e2e globals" block from `eslint.config.mjs` and the `nightwatch.conf.js` entry from the node-globals block below it. Specs are TypeScript with real imports, so no globals injection is needed. - Remove the `@nightwatch/nightwatch-inspector` and `geckodriver` entries from `pnpm-workspace.yaml`'s `allowBuilds`. Do **not** add a `playwright` entry — `@playwright/test` has no postinstall browser download, which is a supply-chain improvement over geckodriver's. - Migrate `tests_output/` → `playwright-report/` + `test-results/` in `.gitignore`, `.prettierignore` and `.stylelintignore` (and fix the stale "nightwatch" comment in `.gitignore`). - Add an explicit exclude for `src/test/e2e/**` in `vitest.config.ts` so the two runners can never collide over a filename. - Scripts: `test:e2e` → `playwright test`; add `test:e2e:ui` and `test:e2e:report`. - Narrow the `pre-push` e2e job in `lefthook.json` to `--project=chromium` so the hook stays fast. - Document the suite in `README.md` / `CONTRIBUTING.md` — neither mentions end-to-end tests today. ### Sequencing Commit in three steps, deletion last, so a reviewer can stop after step 1 with both frameworks working: (1) add Playwright alongside and port the spec; (2) add the new specs (separate issue); (3) remove Nightwatch. ### Out of scope Wiring e2e into CI, and baking browsers into the runner image — both are separate issues, and the image must be baked before CI depends on it. ### Acceptance - [ ] `pnpm test:e2e` passes on chromium, webkit and mobile-safari - [ ] `E2E_FULL_MATRIX=true pnpm test:e2e` also passes on firefox - [ ] `pnpm lint`, `pnpm check`, `pnpm test` pass - [ ] `grep -ri nightwatch` over the working tree returns nothing but the lockfile's unrelated vitest peer entries - [ ] A `--prod` install tree no longer contains `selenium-webdriver`

supernaut lade till etiketterna

area/web

ready-for-implementation

type

chore

2026-08-02 16:06:49 +00:00

supernaut lade till detta till projektet Bitborg Web

2026-08-02 16:06:52 +00:00

supernaut refererade till detta ärende

2026-08-02 16:10:16 +00:00

fix(deps): keep the end-to-end test framework out of the runtime image #179

supernaut refererade till detta ärende från en incheckning