The flake-rate instrument never reached test.yml, so three tiers cannot measure

Context

Four distinct arms went red-then-green-on-rerun in neomjs/neo on 2026-09-21, across three tiers: colors/tearOutMatrix (e2e), data/TreeStoreValueBanding (unit), core/DistEsmWorkspaceBoot (unit), and dashboard/DockTabReorderGesture (component — mine, at 11e429f184 on PR #19029). @neo-opus-ada captured the first three at 12:13; I added the fourth.

A day of sightings has produced no rate, and @neo-opus-ada found why while trying to hand me a different explanation. She had a rule saying concurrent Playwright runs share dev servers and was about to offer it as the common cause — then read the workflows and measured it false: test.yml's job is runs-on: ubuntu-latest, so every matrix entry gets its own fresh VM. That rule is a local hazard and does not reach a hosted runner.

What she found instead is structural, and I verified all four claims at source before filing.

Sweep attestations (2026-09-21T13:5xZ):

• (i) Live latest-open: latest 20 open issues read; nothing equivalent.
• (ii) A2A in-flight: @neo-opus-ada is on #18376 AC-1 and explicitly declined this over me; @neo-opus-vega on Brain #403; @neo-gpt reviewing #19029. No overlapping claim.
• (iii) Memory Core / artifact: #18505 — "No way to measure a flake rate on a hosted runner" (@neo-opus-ada, CLOSED) built this instrument for the e2e tier. This is its unbuilt half, not a duplicate of it.
• (iv) Own-assignment: #19027 (PR #19029), #19033, #19036 (PR #19037). No overlap.
• (1b/1c) Meta-skill / structure map: N/A — a workflow change, no skill and no .mjs created or relocated.

The Problem

test-e2e.yml carries #18505's instrument:

workflow_dispatch:
  inputs:
    specs:      # spec paths to run instead of the tier selection
    repeatEach: # run each selected test this many times — the denominator
concurrency:
  group: tests-${{ github.workflow }}-${{ github.event_name == 'workflow_dispatch' && github.run_id || ... }}

Its docblock states the exact situation this ticket is in: "an intermittent had failures but no DENOMINATOR, and the rerun that unblocks a PR marks the run green and erases the sample it produced." It records #18439's arm reading 2/2, then 1/2, then 1/3, then 0/19 on unchanged source — which is what a measured intermittent looks like.

test.yml has none of it. grep -cE "workflow_dispatch|repeatEach|repeat-each" returns 0. That file owns unit, unit-tz and components — so three of the four arms sit on tiers with no way to produce a hosted denominator at all.

The consequence is not inconvenience, it is that the only available evidence is the one that cannot settle the question. My --repeat-each=5 × 10/10 on DockTabReorderGesture is a local denominator against a CI-only symptom. A rerun is worse than useless as a sample: it overwrites the failure it was meant to measure.

Four sightings, three tiers, zero rates. That is one missing instrument, not four flakes.

The Architectural Reality

• .github/workflows/test-e2e.yml:38-44 — workflow_dispatch with specs and repeatEach; :58 — the dispatch-keyed concurrency group.
• .github/workflows/test.yml — on: carries no workflow_dispatch; owns unit (:46), unit-tz (:52, :56) and components (:60, :64, :68).
• .github/workflows/test.yml:109 — if: ${{ matrix.run == 'true' && steps.head.outputs.current == 'true' }}, where matrix.run resolves from needs.changes.outputs.run_unit / run_components.

That last line is the trap, and @neo-opus-ada named it before anyone could hit it. A workflow_dispatch carries no changed files, so the classifier sets those outputs false, every gated step skips, and the run reports success having executed nothing — the same silent-pass shape as test-e2e.yml's testIgnore printing Skipped: 0. A dispatch must bypass the classifier, and the bypass must be proved rather than assumed.

The Fix

Port #18505's dispatch block to test.yml. It is a copy of a design that has already been measured in production, including the dispatch-keyed concurrency group that exists because two probes cancelled each other.

1. workflow_dispatch with specs and repeatEach, matching test-e2e.yml's input names exactly — a second vocabulary for the same measurement is its own defect.
2. A suite input, because test.yml is a matrix over three tiers where test-e2e.yml is one.
3. Dispatch-keyed concurrency, so two probes do not cancel each other.
4. The classifier bypass, with a proof. On workflow_dispatch, matrix.run must be forced true rather than read from the classifier.

Struck 2026-09-21 by the author during implementation — there is nothing to bypass. classify-test-scope.yml fails open: on a dispatch getChangedFiles() matches neither the pull_request nor the push branch and returns null, so the !files arm sets run_unit / run_components / run_e2e all true with skip_reason: 'changed files unavailable'. The sibling gate, steps.head.outputs.current, returns early with 'true' for any non-pull_request event. Both gates on every step already pass under dispatch, so a bypass would be dead code guarding a door that is open.

@neo-opus-ada's warning is still the reason this was read rather than assumed, and it survives as AC-2: the gate makes "selected nothing" and "everything passed" indistinguishable from outside, so a green dispatch does not satisfy this ticket — the executed count does.

Prescription checked: .github/workflows/test.yml owns the three tiers and the gate. No better owner; test-e2e.yml stays as it is and is the reference.

Contract Ledger Matrix

Decision Record impact: none.

Acceptance Criteria

• test.yml accepts workflow_dispatch with specs, repeatEach and suite, using test-e2e.yml's input names unchanged where they overlap.
• A dispatch executes tests and the receipt proves the count. A run with no changed files reports a non-zero number of executed tests — the failure this AC exists to prevent is a green run that selected nothing, so a green result alone does not satisfy it.
• repeatEach: N on a single spec produces N× that spec's own test count, verified by reading the number off a hosted run rather than inferring it from the input.
• Two concurrent dispatches both reach completion; neither cancels the other.
• pull_request and push behaviour is unchanged — the classifier still gates them, proved by a run on this PR selecting the same tiers it would have before.
• The three intermittent arms each get a hosted denominator recorded on their defect-notes or tickets. Post-merge: the instrument is the deliverable here, the rates are what it is for.

Out of Scope

• Diagnosing or fixing any of the four arms. This builds the instrument; a rate is what lets someone else decide whether an arm is worth a ticket.
• test-e2e.yml, which already has this and stays untouched.
• Making any tier's flake a merge-blocker, or changing rerun behaviour.
• A dashboard, a trend or any storage of historical rates. One dispatch, one number.

Avoided Traps

• Taking the shared-dev-server explanation. @neo-opus-ada had it, offered to hand it over, read the workflows first and killed it — every matrix entry is its own VM. A plausible common cause for four arms is exactly the kind of thing that survives unexamined because it would be so convenient.
• Treating a green dispatch as a working instrument. The classifier gate makes "selected nothing" and "everything passed" identical from the outside. AC-2 requires the count.
• Calling the fourth arm a flake on one green rerun. A rerun erases the sample; it is one observation that the same bytes can pass, which is why this ticket exists rather than a closing comment on #19029.
• A second input vocabulary. Renaming repeatEach here would make the two tiers' measurements incomparable for no gain.

Related

• #18505 — the instrument for the e2e tier (@neo-opus-ada, CLOSED). Its docblock is the design; this is the half it did not reach.
• #18439 — the arm whose 2/2 → 1/2 → 1/3 → 0/19 reading that instrument produced.
• #19031 / #19025 — @neo-opus-ada's intermittents defect-notes, which carry three of the four arms.
• #19029 — where the fourth arm surfaced; it is not that PR's defect and is deliberately not tracked there.

Handoff Retrieval Hints

• query_raw_memories: "flake rate denominator hosted runner repeatEach workflow_dispatch intermittent"
• Reproduce the gap: grep -cE "workflow_dispatch|repeatEach" .github/workflows/test.yml → 0, against test-e2e.yml:38-44.
• The trap to check first: test.yml:109's matrix.run gate against needs.changes.outputs.*.

Origin Session ID: 24396022-ee78-42d8-8935-0c860eb992bc