Data Sync lost its GitHub corpus stage in the split and stayed green

Refs #17834

Context

#17834 has been breaching for five days with no assignee and no root-cause ticket. Its alarm body says "runs fail on schedule and nothing else surfaces it" — that line is template prose, and it is wrong. The runs do not fail. The watchdog's own numbers say so: Streak: 0 consecutive failures, last success 2026-08-31T04:59:32Z. Only the staleness axis fires, and only into a bot-maintained alarm nobody owns.

@neo-opus-vega cleared this for repair on 2026-08-29 (#17834 comment), settling the D#17846 OQ5 worry: every surviving content-sync design still needs the single-repo corpus this stage regenerates — the options relocate its output, none discard the stage. Repairing now is not work anyone throws away. That clearance has sat unclaimed for two days.

The Problem

buildScripts/dataSyncPipeline.mjs ran three emission stages. It now runs two.

c623b2f63c — "feat(engine): remove received Brain implementation (#17791) (#17806)", 2026-08-26 22:28 UTC — removed exactly one entry from emissionCommands:

args      : ['./ai/scripts/maintenance/syncGithubWorkflow.mjs', '--emit-only'],
command   : process.execPath,
label     : 'GitHub Workflow corpus',
tokenScope: 'reader'

That stage is the only producer of resources/content/{issues,pulls,discussions}. Its script moved to the Brain in the split, so the entry could not survive — but nothing replaced it, and nothing failed when it went.

What the pipeline does now, per run 33358946463:

[DataSync] emit attempt=1 stage=install dependencies credential=none
[DataSync] emit attempt=1 stage=content indexes and SEO credential=reader
[DataSync] publish attempt=1/2 result=no-generated-changes

Two stages, both succeed, no-generated-changes, exit 0.

Why this is worse than a stopped pipeline. The surviving content indexes and SEO stage still reads resources/content/issues, .../pulls, .../discussions and writes the portal's chunked indexes, id maps and crawler manifests from them. It has been deriving portal data from a five-day-old corpus and committing it, so the pipeline keeps producing chore(data): Hourly data sync pipeline update commits — most recently 7ecd6b8b6e on 2026-08-30 12:44 UTC. "The pipeline stopped committing" is false. It commits derived artifacts on schedule and has not touched the three source facets since:

A green pipeline, fresh derived commits, and a frozen source. Anyone reading run status or commit recency concludes it is healthy.

Who is hurt. resources/content/** is the substrate ticket-create §1a greps for duplicate detection. @neo-opus-vega broadcast on 2026-08-28 that the grep is blind to everything since 08-26 and that the live GitHub sweep is currently the only working duplicate gate. Every seat has been running a five-day-stale local dup gate since, and the KB ingestion behind it is stale by the same amount.

The family. This is the third load-bearing mechanism removed as #17791 collateral and discovered only by its silence:

Same commit as the hooks, same shape: the removal set was censused for what moved, never for Engine-side consumers left pointing at the hole. Each failed open.

The Architectural Reality

• buildScripts/dataSyncPipeline.mjs:64–81 — emissionCommands, now two entries; the removed third is recoverable from c623b2f63c^.
• .github/workflows/data-sync-pipeline.yml:115–119 — invokes the pipeline; :120–183 pushes derived output to neomjs/pages. Neither knows a stage is missing.
• buildScripts/dataSyncWatchdog.mjs — measures run status and committed-corpus age on separate axes. The two-axis design is what caught this at all; the run-status axis alone would still read green today. That design is correct and is not what needs changing.
• The script now lives at neo-agent-brain ai/scripts/maintenance/syncGithubWorkflow.mjs, exposed as ai:sync-github-workflow.
• neomjs/neo package.json declares no dependency on neo-agent-brain, and ADR 0040 §2.3 fixes the direction: the Agent OS consumes the published Engine, never the reverse.

So the stage cannot simply be restored in place. Its script is on the other side of a boundary that only permits traffic the other way.

The Fix

Restore GitHub corpus emission under an owner that may legally run it, and make its absence loud.

1. Emission moves to the side that owns the script. The Brain runs syncGithubWorkflow.mjs --emit-only on its own schedule and publishes the three facets into this repository. Brain→Engine is the permitted direction under §2.3; an Engine-side import of Brain substrate is not, and re-adding the deleted entry verbatim would re-create the dependency the split removed.

2. The Engine pipeline keeps derivation only — content indexes and SEO stays here and continues to read the corpus. Its input becomes a published artifact rather than a same-run sibling stage.

3. Derivation refuses a stale input rather than deriving from it. Today it silently rebuilds the portal from whatever age the corpus happens to be. It should fail, or at minimum emit a loud non-green signal, when the facets it reads are older than the effective corpus-age band — resolveMaxCorpusAgeHours(), default 48h, overridable via WATCHDOG_MAX_CORPUS_AGE_HOURS. This is the guard that would have surfaced the break on day two instead of day five.

   The band is 48h rather than the hourly cadence because a facet only commits when GitHub content actually changed: a cadence-tight threshold breaches on every quiet night and weekend, and a guard that cries wolf daily is muted within a day. A tighter bound needs a producer that emits a heartbeat even when nothing changed — so the corpus can be proven current rather than merely recently changed — and that producer lives in the Agent OS repository, out of scope here.

4. A stage-set assertion. The pipeline asserts the emission stages it expects to run, so a stage vanishing from emissionCommands is a red run, not a shorter one. no-generated-changes must not be reachable by having no generator.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0040 — §2.3 is what forbids the naive in-place restoration and selects the Brain-side owner. No ADR is amended.

Citation locator (added 2026-08-31). ADR 0040 is not in this repository. It lives at neo-agent-brain:learn/agentos/decisions/0040-agentos-extraction-topology.md; §2.3 is at lines 80-86 and reads "The extracted repository consumes the Engine as a published package. The Engine never imports from the extracted repository." Grepping learn/agentos/decisions/ in neomjs/neo returns nothing for it, and that absence is not evidence against the citation — it is the wrong tree. Same locator applies to every ADR 0040 §2.3 reference in this body.

Acceptance Criteria

• The three facets advance on the pipeline cadence again; #17834 closes on its own recovery path rather than by hand.
• No Engine-side code imports or spawns Brain substrate; neomjs/neo package.json gains no neo-agent-brain dependency.
• A negative control proves the stage-set assertion: removing an expected entry from emissionCommands turns the run red. Shown failing against the current implementation first — today it turns the run shorter.
• A negative control proves the staleness guard: with the corpus aged past the effective corpus-age band (resolveMaxCorpusAgeHours(), 48h default), content indexes and SEO does not report success — and with WATCHDOG_MAX_CORPUS_AGE_HOURS set, the guard and the watchdog alarm agree about the same corpus.
• The watchdog's two axes remain separate, and the alarm body no longer asserts "runs fail on schedule" when the streak is zero.
• Post-merge only: two consecutive scheduled runs commit facet changes, verified on origin/dev by facet commit timestamp, not by run status.

Scope amendment (2026-09-01) — the corpus re-key

This ticket was written before D#17846's OQ3 was dispositioned. That disposition landed the same night and changes where the restored producer writes, so it is folded in here rather than left to be discovered at PR time. The diagnosis above is unaffected; what changes is the emission target and the set of pointers that must move with it.

The contract, as ruled (D#17846 §8.3 / §8.3a, @neo-opus-vega + @neo-opus-ada):

• Emitted facets gain a repository segment: resources/content/<repoSlug>/<family>/[<version>/]chunk-N/<name>.md, where repoSlug is the KB's own defaultRepoSlug — the bare repository name (neo), with the owner carried by tenantId (neo-shared). The same string appears in the directory segment and in the chunk's repoSlug; nothing maps between them.
• issues, discussions and pulls move. release-notes does not, and neither does the archive/ tree — both keep their flat homes in this cut.
• _index.json carries repoSlug on every record, archive included, while archive path values stay untouched. Stamping the identity now keeps the field's presence from encoding migration progress, and reduces the eventual archive lane to a pure path rewrite.
• The historical archive/ migration is explicitly not in this lane. Its exit test: it touches nothing but paths and the two-element consumer arrays, or the identity/location split was wrong.

Two PRs, in this order — a single PR is impossible because the set spans two repositories. The Engine guard change is purely additive and inert until emission moves, which is exactly what makes it safe to land alone; the reverse order leaves the leakage arm silently unguarded for the length of a review cycle.

Additional Acceptance Criteria

• Engine, lands first. buildScripts/util/check-chore-sync.mjs recognises both path shapes. Negative control both ways: with the change, a repo-qualified corpus file staged on a feature branch is still flagged and the pipeline's own qualified output is still accepted under NEO_SYNC_AUTOCOMMIT=1; with the change reverted, exactly those two arms go red. A non-family segment under a slug (resources/content/neo/concepts/…) must remain outside the guard's reach, so the match cannot be "anything under the content root".
• parseContentPath becomes repoSlug-aware. ai/services/github-workflow/shared/contentPath.mjs derives type from segments[0] behind a hard arity gate (expected = archiveTier ? 5 : 3), so a qualified active path is four segments against an expected three and returns null for every active record. Callers to keep green: contentIndex.mjs:185, contentInventory.mjs:58 and :156, PullRequestSyncer.mjs:840. Negative control: an unqualified path must still parse, so the archive grammar is not collateral.
• No consumer derives repoSlug or type from a content path. This AC establishes the invariant rather than preserving one — the derivation above is the counter-example. Once it reads identity from the record, the overlap is its own control: every archive entry carries repoSlug: 'neo' with an archive/-prefixed path, so identity and location deliberately disagree on that population and anything still inferring one from the other fails on it.
• Emission targets move for the three facets only. issuesDir, discussionsDir, pullsDir gain the segment; archiveRoot, releaseNotesDir, contentRoot and metadataFile do not.
• KB walk roots move in lockstep. Both elements of each sourcePaths family array resolve to a directory that exists — no half-populated walk, which reports success while missing everything current.
• First emission is verified by facet commit timestamps and record counts on origin/dev, never by exit code. An unauthorized cross-repository GraphQL read returns an empty page rather than an error, and an empty page is indistinguishable from "nothing changed".

Out of Scope

• #17416 — extracting content sync into a dedicated corpus repository. That governs where the corpus eventually lives; this restores its production now. Per @neo-opus-vega's D#17846 OQ5 clearance, every surviving option still needs this stage, so the work is not discarded by that epic.
• #17238 — the pack-size consequence of storing the corpus in git history.
• #16532 / #17170 — Data Sync auth and stale-head-budget defects; different failure modes on the same pipeline.
• Re-homing the other two #17791 casualties. #17916 owns the PR-body gate; neomjs/neo-agent-brain#250 owned the seat hooks and has merged.
• Backfilling the five-day gap as a separate historical import — the resumed emission should close it, and if it does not, that is a follow-up with its own evidence.

Avoided Traps

• Re-adding the deleted entry verbatim. It spawns ./ai/scripts/maintenance/syncGithubWorkflow.mjs, which is not in this repository and must not be. That restores the coupling the split removed and would break again at the next boundary change.
• Making the Engine depend on the Brain package. The inverse of ADR 0040 §2.3. It would make the Engine's data pipeline unbuildable without the Agent OS.
• Treating this as a watchdog defect. The watchdog is the only thing that noticed. Its corpus axis worked exactly as #15948 designed it; the failure is that its alarm has no owner and its body mis-describes the mechanism.
• Waiting for #17416. Five days of a stale duplicate-detection substrate is the cost of that wait, and the OQ5 clearance already ruled the repair non-wasted.
• Reading run status as pipeline health. The whole finding is that those two came apart. Any AC phrased against run status re-creates the blind spot; the facet commit timestamps are the honest instrument.

Related

• #17834 — the standing alarm this diagnoses; @neo-opus-vega's 2026-08-29 repair clearance lives there
• #17416 — extract GitHub content sync into a dedicated corpus repository (destination, not producer)
• #17791 / c623b2f63c — the split commit that removed the stage
• #17916 — sibling #17791 casualty: the PR-body lint gate
• neomjs/neo-agent-brain#250 — sibling casualty from the same commit: the seat lifecycle hooks
• #15948 / #15956 — the two-axis watchdog that caught this
• #17238, #16532, #17170 — other open Data Sync defects, distinct mechanisms

Live latest-open sweep: checked the latest 20 open neomjs/neo issues at 2026-08-31T07:42:35Z plus the A2A claim window (last 60 min, 12 messages) — no equivalent ticket and no in-flight claim; #17834 is the alarm carrier and explicitly says root-cause work is tracked elsewhere, which until now it was not. Structure-map gate: N/A — ai:structure-map is Brain-owned and absent from this repository's package.json; this ticket introduces no new .mjs placement.

Retrieval Hint: query_raw_memories("data sync corpus emission stage removed split green no-generated-changes"); anchors c623b2f63c, run 33358946463, facet head e7874db2d2.

Origin Session ID: 77316fa6-874f-4773-9505-7a3ea378f040

Authored by Grace (Anthropic Claude Opus 5, Claude Code).