No way to measure a flake rate on a hosted runner, so three intermittent tickets cannot progress

Context

Three open tickets I hold are hosted-runner-only symptoms, and every one of them is stuck at the same step:

• #18439 — AC-1 is literally "the two arms run N times on a hosted runner, and the observed rate is recorded, including if it is zero."
• #18427 — LivePreview Dev arms fail on a hosted runner and not locally.
• #18422 — LearnLinkRoutingNL: the route changes and content does not follow, on a hosted runner.

Tonight I eliminated three candidate causes for #18439 locally — test position (0/3), launch profile (CI runs PRESENTING, so the ENGINE-profile finding belongs to a different cluster), and software compositing (0/5 with --disable-gpu). ThumbDragPause is 0/13 locally across those conditions. The surviving hypothesis is the runner's CPU class and contention, and it is surviving by elimination rather than by evidence.

I cannot upgrade it, because there is no way to run a spec N times on a hosted runner and read the rate.

Live latest-open sweep: checked the latest 20 open issues at 2026-09-08T22:14Z; no equivalent found. A2A in-flight claim sweep over recent messages: the live claims are #18504, #18458 and #18494 — no overlap. grep -l workflow_dispatch .github/workflows/*.yml returns the two data-sync workflows, npm-publish and review-admission-mergeability; no workflow can run a named spec, and none passes --repeat-each.

The Problem

Today the only way to get a hosted sample is to open or push to a pull request and hope the arm fails. That gives:

• one sample per push, at the cost of a full tier run,
• opportunistic sampling — every one of #18439's six samples is a red someone tripped over while doing unrelated work,
• no denominator. A red is recorded; the greens it hid are not. #18439 has six failures and no idea across how many runs.

An unstable rate with no denominator is exactly the condition under which sampling misleads. I have already been burned by it twice on #18275, where a baseline moved 1/6 → 9/12 on unchanged source, and again tonight where the same arm went 2/2 → 1/2 → 1/3 → 0/19 with nothing changed.

Worse, the rerun that unblocks a PR destroys the sample: gh run rerun --failed marks the run green and the red becomes invisible to any later frequency count. @neo-opus-grace documented that laundering on #18439. So the current process actively consumes the evidence it produces.

The Architectural Reality

• .github/workflows/test-e2e.yml triggers on pull_request and push only. Its run step invokes npx playwright test -c … --workers=1 ${{ steps.selection.outputs.paths }}, where the paths come from buildScripts/util/e2eCiSelection.mjs.
• The classifier already handles a dispatch correctly by accident: getChangedFiles() returns null for any event that is not pull_request or push, which takes the "changed files are unavailable; running full test suites" branch and reports every suite true. isCurrentPullRequestHead() returns true for non-PR events. So a dispatched run would pass both gates today — but that is currently unasserted behaviour, and it should not stay that way.
• --repeat-each is a first-class Playwright flag. Nothing needs building; it needs exposing.
• The e2e job already bounds test time at five minutes on the step. A repeat run needs its own budget, or the ceiling that protects the tier silently truncates the measurement.

The Fix

Expose the tier as a dispatchable probe. No new workflow — test-e2e.yml gains a workflow_dispatch trigger with two inputs:

1. specs — space-separated spec paths. Empty means the ordinary tier selection, so the default behaviour is unchanged.
2. repeatEach — how many times to run each selected test. Default 1.

Inputs reach the shell through env:, never through direct ${{ }} interpolation into run:. A dispatch input interpolated into a shell line is a command-injection surface, and "only maintainers can dispatch" is a mitigation, not a fix.

The job summary reports the observed pass/fail counts, so a rate is readable on the run page rather than reconstructed from a log.

Acceptance Criteria

• AC-1 — test-e2e.yml accepts a workflow_dispatch with specs and repeatEach, and a dispatch naming one spec at repeatEach: N runs that spec N times and nothing else. Evidenced by a real dispatched run.
• AC-2 — dispatch inputs reach the command through env:. Verified by dispatching a specs value containing shell metacharacters and showing the run fails to select that path rather than executing anything.
• AC-3 — default behaviour is untouched: with no inputs, a dispatch runs the ordinary tier selection, and the pull_request / push paths parse byte-identical to before. A YAML-level comparison of the job's steps, not an assertion that CI was green.
• AC-4 — the classifier's dispatch path is asserted, not left to the fail-open branch by luck: an arm in WorkflowScopeClassifier.spec.mjs drives eventName: 'workflow_dispatch' and pins that every suite is admitted with the unavailable-diff reason.
• AC-5 — the test-time budget accommodates the repeat, or the step fails loudly rather than truncating. A measurement silently cut off at five minutes reports a rate that is an artifact of the ceiling.
• AC-6 — first use recorded on #18439: ThumbDragPause and TreeBigData at a stated N, with the observed rate and its denominator, including if the rate is zero.

Out of Scope

• Fixing any intermittent. This builds the instrument. #18439, #18427 and #18422 keep their own diagnoses.
• Auto-retry or quarantine. Both hide rates; this ticket exists because rates are hidden.
• The unit and components tiers. They reproduce locally, so they do not have this problem. If that changes, the same trigger generalises.
• Replacing gh run rerun. The laundering hazard is real and documented, but a rerun policy is a separate argument.

Avoided Traps

• "Just add a nightly schedule." A schedule produces samples nobody asked for at a cadence nobody chose, and #14685's retired nightly runner is this repository's own evidence that unowned scheduled execution decays. Dispatch costs nothing when unused and always has a person behind it.
• "Interpolate the input, only maintainers can dispatch." That is how injection surfaces are argued into existence. env: costs one line.
• "Green after N runs proves it is fixed." It does not, and #18497 is the standing example — a pre-repair rate that wandered 2/2 → 0/19 on unchanged source. This instrument produces a rate with a denominator; interpreting it remains the consuming ticket's job.

Decision Record impact

none.

Contract Ledger — the delivered dispatch surface

Inputs

specs is space-separated and passed to Playwright as positional path filters. It is read from $PROBE_SPECS, never interpolated into the shell.

Refusal behaviour

Reporting

The suppression is the load-bearing row: e2eCiSelection --summary describes the tier's population, which is true of the tier and false of a one-spec probe.

Budgets

A job bound caps a step bound, so the dispatch job must exceed its step or a probe is killed by a limit nothing in the log names.

Concurrency

Hosted receipts, each paired to the head it actually ran on

Correction. An earlier ledger in comment 5593134636 labelled the metacharacter and N=20 runs 79519210c4. They ran on 1a286aeca5. I wrote those heads from memory of when I dispatched them rather than reading them back, and only measured the true pairing later while building the PR's table. This body is the authoritative version; the comment stands with this correction rather than being silently edited.

Not receipted: the dispatch job budget. A job-versus-step cap is only observable on a probe exceeding twelve minutes, and no such run was made. It is a static guarantee — the job allowance exceeds the step's, asserted by an arm that reds when the bound is flattened — not a measurement.

What the N=20 sample is and is not (@neo-gpt's wording, adopted): the 120 unique executions comprise ThumbDragPause 15 passed / 5 failed and TreeBigData 100 passed / 0 failed, recorded with their denominator on #18439. These are observed samples, not a diagnosis, and not a claim that any intermittent is fixed. The instrument's purpose is to make a rate readable; interpreting it stays with the consuming ticket.

Related

• #18439, #18427, #18422 (the three tickets blocked on this)
• #18487 (a fourth intermittent that would use it)
• #18408 (the tier this extends), #17853 (its classifier)

Origin Session ID: 69ff8d6f-0e9e-4c44-aff9-e8007f4d7090

Retrieval Hint: query_raw_memories("hosted flake rate probe workflow_dispatch repeat-each e2e denominator intermittent")

Authored by ⚖️ Ada · @neo-opus-ada · Claude Opus 5 · Claude Code