e2e-brain shards split by test count, so one shard nears the ceiling

Context

Operator, 2026-10-23 ~18:50Z, watching #20074's checks: e2e-brain (7/10) had been running for 8 minutes, "above our limit". The tier's limit is the five-minute test-step ceiling (.github/workflows/test-e2e-brain.yml:54: "the test step itself enforces the five-minute ceiling, as the engine tier does"). test-e2e.yml:11 states the policy behind it: "A shard nearing it gets a sibling, never a higher ceiling: the ceiling is the wait every pull request pays."

Sweeps:

• Live latest-open sweep: the latest 20 open issues at 2026-10-23T19:02:56Z hold no equivalent.
• A2A in-flight sweep: no claim on this surface in the 30 newest messages.
• MC sweep: query_raw_memories on "e2e-brain shard takes too long … five-minute ceiling exceeded", 8 results, no prior decision on shard balance. Prior art: #19078 (the engine tier, same symptom, fixed with a third shard) and this tier's burn-in under #19655 (#19657, #19675: ten shards).
• Own-assignment sweep: 2 open (#19919, #15000), none overlapping.

The Problem

Where the 8 minutes go. Run 40005321524 (#20074 at 0fb6cc852b, full tier, all green), shard 7, measured from run creation:

Three of the ten shards waited 79–108 s for a runner on that run.

The test step is unbalanced. The step times on that run ranged from 48 s (shard 6) to 256 s (shard 8, 85 % of the ceiling). Shards held 4–19 files and 6–58 tests each. Playwright's --shard takes no duration input, so shard 8 carried WorkstationNL.spec.mjs (71 s) and WorkstationPopupTabDragNL.spec.mjs (40 s) together.

What grew is the total, not the slowest shard. Step durations were sampled across 61 successful full-tier runs, every sixth run per day from 10-12 to 10-23, read from the GitHub Actions jobs API for the Run the Brain-gated tier step:

• The slowest shard's step stayed at 223–276 s throughout. The margin was thin from the first blocking day; this is not a fresh regression.
• The summed step time rose from 1,337–1,365 s (10-12/13) to 1,577–1,765 s (10-23), about +25–30 %.
• *.NL.spec.mjs files on origin/dev (git ls-tree over test/playwright/e2e) went from 82 at 1ea2a8b8fc (10-12) to 107 at 395f4c83dc (10-23), 18 of them since 10-20. The run above executed 118 files and 347 tests (shard receipts).

As the total keeps rising, count balancing will sooner or later put two heavy files in one shard and push it over the ceiling.

The heaviest files on that run, per-test list-reporter durations summed per file:

The Architectural Reality

• .github/workflows/test-e2e-brain.yml:
  • The matrix is shard: [1..10] / shards: [10] (:60-63); the job timeout-minutes is 12 (:55) and the step's is 5 (:109).
  • Each shard runs npx playwright test … --shard="$SHARD" --retries=… --test-list-invert <quarantine> $paths (:125), where $paths is e2eCiSelection.mjs --brain-paths.
• test/playwright/playwright.config.e2e.mjs:82-84 sets fullyParallel: false, workers: 1 and timeout: 90000.
• buildScripts/util/e2eCiSelection.mjs already decides the tier's population (--brain-paths), quarantines (--brain-test-list) and retries (--brain-retries). It is the one place that can also decide each shard's files.
• Report the shard's receipt already reports files and tests per shard.

The Fix

Keep ten shards and balance them by measured duration instead of count:

1. Add a committed per-file duration table for the Brain tier, in seconds, with a default for files it does not know yet.
2. e2eCiSelection.mjs returns shard N's files from a longest-processing-time assignment over that table. The assignment is deterministic, with ties broken by path.
3. The workflow passes that file list to npx playwright test and drops Playwright's --shard.
4. The receipt prints per-file durations in the table's format, so the table can be refreshed instead of going stale silently.

A simulation over that run's 118 file durations puts the slowest of ten duration-balanced shards at ≈ 144 s of test time (≈ 165 s of step), against 256 s today. Twelve count-balanced shards would add two runners to every PR on a day when shards already wait up to 108 s for one.

Acceptance Criteria

• AC-1: e2eCiSelection.mjs assigns the Brain tier's files to shards by duration. Unit arms pin that the assignment is deterministic, that every selected file lands in exactly one shard, and that an unknown file gets the default.
• AC-2: on a full-tier PR run, the slowest Run the Brain-gated tier step is ≤ 200 s and the spread between the slowest and fastest shard is ≤ 60 s.
• AC-3: the selected population runs exactly once: summed across shards, the receipts' file count equals --brain-paths, and the quarantine inversion still applies.
• AC-4: the table's refresh path is documented beside the table, and learn/guides/testing/WhiteboxE2E.md says how the tier is sharded.
• AC-5 (post-merge): five consecutive full-tier runs on dev pushes keep the slowest shard's step ≤ 200 s.

Out of Scope

• Runner queue wait (up to 108 s above) and per-shard provisioning (~66 s). They trade against each other (fewer, longer shards; caching the Brain install) and are worth a follow-up once the test step is balanced.
• Raising the five-minute ceiling.
• Making individual specs faster.
• The engine tier (test-e2e.yml), which has its own shards.

Avoided Traps

• More count-balanced shards: the precedent's fix, #19078. Here it costs runners on a contended day, and heavy files still collide.
• A higher step timeout: the ceiling is the wait every pull request pays.
• Durations read live from an earlier run's artifacts: that ties a PR's sharding to another run's state. A committed table is reproducible.
• More workers per shard: the NL fixture's per-worker Bridge and windows put that outside this ticket's measurement.

Related

Parent #19655. Precedent #19078 / PR #19079. This tier's ten-shard burn-in: #19657, #19675. Observed on #20074.

Origin Session ID: cbc656fa-c6fc-4640-86fc-0aca97280d47

Retrieval Hint: "e2e-brain shard duration five-minute ceiling count-balanced sharding runner queue provisioning"