The e2e suite runs in no CI workflow

Context

Operator, 2026-08-29, on PR #17852: "the regression bug sneaked in today, with GREEN unit tests. we need component based or e2e testing that PROVES that parent menus get closed correctly. unit tests without a main thread are not enough."

Chasing that produced a second, wider finding that PR does not own: no CI workflow runs the e2e suite. npm run test-e2e exists, 83 specs sit behind it, and nothing on pull_request or push calls it.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-29T16:21Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages (all read-states): no overlapping [lane-claim].

The Problem

.github/workflows/test.yml is the only workflow that runs Playwright. Its matrix is exactly two entries:

include:
  - suite: unit
    run: ${{ needs.changes.outputs.run_unit }}
  - suite: components
    run: ${{ needs.changes.outputs.run_components }}

The classifier job emits run_unit and run_components and nothing else. npm run test-e2e is defined in package.json and invoked by no workflow in the repository.

Measured at dev, 2026-08-29:

The 15 non-gated specs are the sharp end: they need nothing but a checkout and a browser, and they still never run on a PR. They include e2e/grid/RowPinning.spec.mjs, e2e/grid/Tree.spec.mjs, e2e/core/DragSensorDispatch.spec.mjs, and e2e/rendering/LivePreviewMultiWindow.spec.mjs — multi-window and drag-sensor surfaces with no equivalent coverage in the two tiers CI does run.

How this surfaced

e2e/rendering/ContextMenuNL.spec.mjs asserts that a context-menu leaf click leaves zero menus standing. That is precisely the defect #17850 describes, and the spec landed in the same commit series. It never ran on the PR that regressed the behaviour, because it never runs on any PR.

This ticket is not "and it would have caught it." It would not have. Measured separately (PR #17852): that spec's assertion is expect(page.locator('.neo-menu-list')).toHaveCount(0), which is web-first and re-polls until the timeout; on the regressed source the tree does reach zero via a later focus round-trip, so the spec passes on the defect. Both holes are real and independent, and only the assertion-shape one is fixed there. This ticket owns the other.

The Architectural Reality

• .github/workflows/test.yml:230-234 — the two-entry matrix. Every step gates on matrix.run, so a third entry is additive rather than a restructure (#15368 generalised the per-suite flag for exactly this reason).
• .github/workflows/test.yml:194 carries the precedent verbatim: "a missed path silently skips the suite (#15368 — this suite ran in NO workflow at all until it was wired here)." That comment is about the components suite. The same sentence is now true of e2e, and the classifier's whitelist has no e2e-relevant path predicate at all.
• test/playwright/playwright.config.e2e.mjs — discoverExternalBrainSpecs() testIgnores every neuralLink-importing spec unless NEO_AGENTOS_RUNTIME_ROOT names an absolute Brain checkout. That is correct fail-closed behaviour and is why a first cut can run the 15 without provisioning anything.
• The e2e config sets trace: 'on', a 90s timeout, workers: 1, reuseExistingServer: false, and pins channel: 'chrome' rather than Playwright's bundled Chromium. Runtime cost and the branded-Chrome dependency are the two real design questions, not the wiring.
• Some e2e specs are benchmarks (e2e/benchmarks/), whose numbers are meaningless on shared runners. Suite composition is part of the decision, not an afterthought.

The Fix

Wire an e2e suite entry into test.yml, scoped deliberately rather than wholesale:

1. Add run_e2e to the classifier with its own path predicate (src/, examples/, test/playwright/e2e/, test/playwright/playwright.config.e2e.mjs, test/playwright/fixtures.mjs, dep manifests, the workflow). examples/ matters here and is absent from the components whitelist, because e2e specs drive example apps.
2. Add a third matrix entry that runs only the non-Brain-gated set — no NEO_AGENTOS_RUNTIME_ROOT, so discoverExternalBrainSpecs keeps the other 68 ignored by construction. No new secret, no runtime provisioning.
3. Decide benchmark disposition explicitly: exclude e2e/benchmarks/ from the CI project, or keep them and state that their numbers are not a gate.
4. Record the measured wall-clock of the resulting set on this ticket. If it is not affordable, say so with the number and cut the set — do not wire a suite whose cost nobody wrote down.
5. Install Chromium for the new suite the way the components entry already does, or record why branded Chrome is required.

Explicitly not proposed: making the job required. dev has zero required status contexts today (#17783 R1) — every workflow is advisory. Binding contexts is that ticket's concern, and wiring a job that runs is a prerequisite for it, not a substitute.

Acceptance Criteria

• test.yml runs a third suite entry that executes e2e specs on pull_request and push to dev.
• The classifier gates it on an e2e-relevant path predicate that includes examples/, verified by a PR touching only examples/** reporting run_e2e=true.
• The executed set is the non-Brain-gated one: a run log shows the NL-gated specs ignored (not failed, not skipped-with-error) and a non-zero count of specs actually executed.
• The suite's wall-clock is recorded on this ticket from a real CI run.
• Benchmark specs have a stated disposition — excluded, or included with an explicit no-gate note in the workflow.
• A deliberately reverted src/ change makes the new job go red, proving it gates rather than merely reports. Green alone is not evidence the wiring is live (#15368 is the anchor: the components suite existed and ran nowhere).

Out of Scope

• Running the Brain-gated 68 in CI. Runner topology for the external Brain runtime is a separate decision. #17844 owns battery execution of that set from maintainer seats with a checkout; @neo-fable proved the two-clone, one-env-var path works from any Engine seat (2026-08-29).
• Making any check required on dev. #17783 R1.
• Moving or deleting specs. #17422 owns relocating the devindex-driven e2e specs out of this repo; whatever it moves, this wiring follows.
• Authoring new e2e specs. Feature leaves keep their own tiers.
• Assertion-shape review of the existing specs. PR #17852 fixes the one spec measured blind. A sweep for other settling assertions over transient behaviour would be its own ticket — worth one, and not this one.

Avoided Traps

• "Just run the whole e2e suite in CI." 68 of 83 specs would be silently ignored for want of a Brain checkout, and a green job that skipped 82% of its specs reads as coverage it does not have — the same false-green shape this ticket exists to remove.
• "It would have caught #17850." It would not have; measured. Wiring CI and fixing the assertion are two independent repairs, and conflating them would have closed this with the PR.

Decision Record impact

none.

Related

• #17850 · PR #17852 (origin; fixes the assertion-shape half)
• #17783 (required status contexts on dev — the binding half)
• #17844 (battery execution of the Brain-gated set)
• #17422 (relocating devindex-driven e2e specs)
• #15368 (the components suite's identical prior instance)

Origin Session ID: 57d042dc-6295-4fea-8347-a79adb8135fc

Retrieval Hint: query_raw_memories("e2e suite runs in no CI workflow test.yml matrix unit components test-e2e 83 specs")