LearnNewsExamplesServices
Frontmatter
id18408
titleGive the engine e2e tier a CI runner that reports what it covers
stateClosed
labels
enhancementaitestingbuild
assigneesneo-opus-ada
createdAtSep 7, 2026, 1:06 AM
updatedAtSep 7, 2026, 1:47 AM
githubUrlhttps://github.com/neomjs/neo/issues/18408
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtSep 7, 2026, 1:47 AM

Give the engine e2e tier a CI runner that reports what it covers

Closed Backlog/active-chunk-21 enhancementaitestingbuild
neo-opus-ada
neo-opus-ada commented on Sep 7, 2026, 1:06 AM

Context

Split from #17853 so that ticket can stay open on its remaining criterion while the delivered half lands.

#17853 establishes that the e2e tier runs in no CI workflow — 104 spec files, 60 of them dock or multi-window, executed by nothing. Its six ACs have since been ruled on by its author, @neo-opus-grace (ruling): AC-1 amended and met, AC-3 through AC-6 met, and AC-2 — a classifier path predicate — deferred as a follow-up with a named home, because building a separate workflow multiplied its cost from two lines inside test.yml to either duplicating ~200 lines of classifier or extracting a reusable one. Grace's words: "#17853 stays open on AC-2. I am not closing it on the merge."

PR #18404 therefore cannot carry Resolves #17853 — merging it would auto-close a ticket both its author and I intend to keep open. This ticket is the leaf that PR actually delivers.

Live latest-open sweep: checked the latest 12 open issues at 2026-09-06T23:06Z; no equivalent. A2A claim sweep over tonight's nightshift: #18406 (@neo-gpt-emmy), #18400 (@neo-gpt), #18401 (@neo-opus-grace) — none on e2e CI wiring.

The Problem

No workflow calls playwright.config.e2e.mjs. Not gated — never called. So an e2e arm's green is someone ran it by hand once, on some tree, and an arm can rot for weeks invisibly.

Two structural limits bound what CI can run, both established empirically rather than assumed:

  • GPU — specs booting the workstation app inherit animated OffscreenCanvas work no hosted runner composites.
  • Time — @tobiu's ceiling is five minutes of test time, and the full Brain-free set takes 7.2m serial.

A third bounds the reporting: testIgnore DESELECTS the Brain-dependent specs rather than skipping them, so a run missing four fifths of the tier still prints Skipped: 0. A naive wiring buys a green badge certifying 14 of 104 files with the rest invisible — a worse false green than the honest absence it replaces.

The Fix

A dedicated E2E (engine tier) workflow running the canvas-free, build-independent selection serially under NEO_TEST_SKIP_CI=true, with buildScripts/util/e2eCiSelection.mjs as the single source for both the invocation and the coverage report so the two cannot describe different populations.

Acceptance Criteria

  • A workflow executes e2e specs on pull_request and push to dev.
  • The test step is bounded to five minutes, separately from provisioning.
  • The job summary states the fraction covered and names every excluded population — Brain-deselected, GPU, build-dependent, and selected-but-quarantined — each with an owner.
  • The selection floor fails the job when a path goes stale or an exclusion is undeclared, at the invocation boundary rather than after the runner starts.
  • A controlled src/ mutation turns the tier red, proving it gates rather than reports.

Out of Scope

  • The classifier path predicate — that is #17853's remaining AC-2, and its correct implementation is a reusable-workflow extraction rather than a second copy of the predicate.
  • The Brain-dependent tier (84 files) and the GPU tier (4 files).
  • Repairing the excluded and quarantined arms; they carry owners, not fixes, in this delivery.

Related

  • #17853 — the parent finding, staying open on AC-2.
  • PR #18404 — the delivery.

Origin Session ID: 95f9ff6e-1ba5-43b6-860a-208fbf7f4442