Skip to content

fix(pm): mask the fixture builders only a self-test can reach - #13930

Merged
os-project-manager merged 3 commits into
mainfrom
claude/issue-13781-mask-selftest-helpers
Aug 31, 2026
Merged

os-project-manager merged 3 commits into
mainfrom
claude/issue-13781-mask-selftest-helpers

Conversation

@os-project-manager

@os-project-manager os-project-manager commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #13781

The defect

maskSelfTests blanks the brace-balanced body of every declaration whose own name matches SELF_TEST_DECL. A fixture builder that only the self-test calls is named for what it builds — makeSource, buildFixtureTree, stampFor, fixtureCommit — so it never matched, its body survived the mask, and its fixture path literals reached extractWatchHints as if they were paths the gate opens.

Premise re-derived on this branch's base 73c846687 (the card measured it at 4301f7846; still live). The card's own command:

node -e "import('./scripts/pm/dispatch-gates.mjs').then(async m => {
  const src = require('node:fs').readFileSync('scripts/pm/release-rehearsal-clone.mjs','utf8');
  console.log(m.extractWatchHints(m.maskSelfTests(m.maskComments(src)), 'scripts/pm/release-rehearsal-clone.mjs'));
})"
before (73c846687)                            after (this branch)
[ 'scripts/pm/release-rehearsal-clone.mjs',   [ 'scripts/pm/release-rehearsal-clone.mjs',
  '.changeset/*.md',                            '.changeset/*.md',
  '.changeset',                                 '.changeset' ]
  '.changeset/config.json',
  '.changeset/README.md',
  '.changeset/one.md',
  '.changeset/two.md' ]

The four that go are written by makeSource (L485) into a temp git repo; the three that stay are the population the file really declares.

The shape

In-file reachability over top-level declarations, bounded to one file, with the mask applied to callables only.

A top-level function/class is masked when it is reachable from a self-test body and not reachable from anything else the module does. The roots of "anything else" are the module-body statements outside every declaration (imports, top-level side effects, export { … } lists, the entrypoint guard at the bottom) plus every exported declaration, which is reachable from outside the file by definition. A self-test body is reached but never traversed — these scripts call selfTest() from module scope, so traversing it would make every helper root-reachable and the predicate vacuous. The second conjunct is the safety half: a helper shared by the self-test and the real gate body stays unmasked.

Two mechanics that are not decoration:

  • The parameter list is skipped before brace counting, and this repairs live files. A destructured default puts braces in the signature, and counting those closes the body before it opens. ⚠ An earlier revision of this PR and of the docblock called that bug latent — "no self-test in this tree takes a parameter (61 of 61 are selfTest())". That was false, and the correction makes the fix worth more, not less. Three self-test entry points under scripts/ carry a brace in their signature on origin/main today:

    scripts/check-test-completeness.mjs:576            function selfTest({ quiet = false } = {})
    scripts/measure-position-name-fold-census.mjs:689  function selfTest({ quiet = false } = {})
    scripts/workspace-enumerator.mjs:328               export function selfTest({ root = null } = {})
    

    Bytes maskSelfTests changes in each file, this module against a staged copy of the one on origin/main (6193e576d; the three subject files and all three of the module's relative deps are byte-identical between that sha and this branch's base, so the staged copy is an exact control):

    file                                            main    branch
    check-test-completeness.mjs                       30    14848
    measure-position-name-fold-census.mjs             30     3961
    workspace-enumerator.mjs                          34     6294
    check-empty-changeset.mjs  (control, no brace)  48044    48044
    

    Thirty bytes is the destructured parameter and nothing else — the entire self-test body was surviving the mask. The control blanking an identical 48044 both ways is what makes those three a reading rather than an artifact. It did not move the census, and that is luck rather than design: extractWatchHints returns identical sets over the three ([], [], and the same 8 hints) because their self-test bodies happen to carry no path literal their module bodies do not already carry. A fourth file with the same signature shape and one fixture path in it would have been a live fabricated lead.

    The neighbouring 61 of 61 at column 0 claim was re-derived rather than assumed: across scripts/ at 6193e576d there are 169 carrier files and 185 declarations (111 function selfTest(, 28 export function selfTest(, 19 async function selfTest(, 8 export async function selfTest(, 19 compound names) — the counts were stale by a factor of three, but the two properties the anchor rests on both hold: 185 of 185 at column 0, and a column-0 scan for the const/arrow spelling finds zero. Both docblock paragraphs now say that, and the paragraph tells the next reader to re-derive the counts rather than quote them. One measured aside recorded there: this module's own maskSelfTests matches SELF_TEST_DECL, so the file blanks that function's body when it scans itself — true long before this change, and free only because none of the functions it reaches spells a path.

  • A reference inside a template interpolation is a reference. scan.interpolation is consulted, and skipping it is not academic: release-rehearsal-clone.mjs names its own path constant only from inside template literals, so a scan that read ${SELF} as string text found SELF unreferenced and masked away the one hint that file really declares. Measured — it was the first version's behaviour.

A declaration whose span cannot be closed is dropped rather than run to end of file, so one unterminated span cannot swallow the declarations after it. The mask-to-end-of-file behaviour a malformed self-test has always had is kept where it lives.

Invariant: the new mask is a strict superset of the old one. Verified positionwise over all 204 scanned scripts: 0 violations, and 158 of the 204 mask byte-identically to origin/main.

Falsification: fleet-wide hint census

Population: every file any discovered family names, plus every first-party module those files import — the exact set discoverFamilies runs extractWatchHints over. Same tree, same command, only scripts/pm/dispatch-gates.mjs differing.

                     origin/main (73c846687)   this branch
files scanned                          204            204
families                               193            193
per-file hints                        1058            954     (-104)
per-family hints                      1282           1179     (-103)
files whose hint set changed             9
hints GAINED                             0

Covering relation over all 7621 tracked files, per family: 1 covered pair lost, 0 gained. Adjudicated below.

Every removed hint, by the declaration it came from

21 newly masked declarations in 9 files. No removed hint is unattributed.

file newly masked declarations hints removed
scripts/pm/release-rehearsal-clone.mjs runSelf L458, fixtureCommit L470, writeFile L475, makeSource L485, cloneOf L516 .changeset/one.md, .changeset/two.md, .changeset/config.json, .changeset/README.md — all four written by makeSource into a temp repo
scripts/check-test-source-alias.mjs fixture L1950, buildFixtureTree L1963 44 sandbox package/source paths (packages/clocked-decoy, packages/violator, src/thing.ts, …) built under a tempdir
scripts/check-type-source-resolution.mjs fixture L1256, buildFixtureTree L1273 40 sandbox paths (packages/star-trap, packages/unparseable, spec/dist/*, …)
scripts/check-skills-token-ratchet.mjs fixtureTree L547 skills/README.md, skills/demo/SKILL.md, skills/demo/rules/nested.md, skills/demo/references/_index.md, skills/demo/evals/deep/deeper/case.md, skills/not-a-skill/notes.md
scripts/check-agent-test-spelling.mjs makeFixtureTree L791, baseFixtureFiles L802 .claude/agents/os-dev.md, .github/workflows/lint.yml, scripts/keep.sh, skills/x/SKILL.md — names written into the fixture tree, not read from the repo
scripts/check-whole-set-label-write.mjs writeTree L711, baseFiles L720, withTree L811 .github/actions/setup-pnpm/action.yml, .github/workflows/other.yml
packages/spec/scripts/check-entry-nameability.ts writeFixture L452 packages/spec/scripts/dist/other.d.ts, packages/spec/scripts/dist/other.js
scripts/check-undeclared-dep-imports.mjs fixture L700, makeTree L706 packages/subject/package.json
scripts/check-console-injection.mjs tmpdir L452, makeSpecPkg L458, makeUnbuiltSpecPkg L477, makeDist L490, stampFor L503 scripts/assert-console-spec-injection.mjs — the generatedBy field of a fabricated stamp object

Three of these read like real paths and are not:

  • .claude/agents/os-dev.md / .github/workflows/lint.yml (check:agent-test-spelling) are filenames the fixture tree creates under a tempdir. The family's surviving hints still cover both files — 0 covered pairs lost.
  • .github/actions/setup-pnpm/action.yml (check-whole-set-label-write.mjs): same, 0 covered pairs lost.
  • scripts/assert-console-spec-injection.mjs (check:console-injection) leaves that file's own hint set empty, but the family keeps the lead through a better-provenanced channel: it is inherited from the imported module scripts/console-spec-probes.mjs, with hintOrigin naming it. The family's hint set is unchanged.

The one covered pair lost, adjudicated

skills/README.md stops being covered by scripts/check-skills-token-ratchet.mjs. That gate's own source declares the file outside its population, in prose and in a pinned case:

 *   3. OUTSIDE — `skills/README.md`. Not inside any published skill directory:
 * is skipped whole; `skills/README.md` is not inside a skill directory and never
   ['skills/README.md is outside the population (population 3)', walked.includes('skills/README.md'), false],

The hint came from fixtureTree (L547), which writes skills/README.md into a temp tree precisely to prove the gate ignores it. A card editing skills/README.md was being told to run a ratchet that provably does not read it. That is the defect, not a cost of the fix.

Zone 2, measured

  • A (banner anchor disfavoured) — CONFIRMED, and adopted rather than re-litigated. SELF_TEST_DECL's docblock argues the general case ("a declaration is a thing the language guarantees, a marker comment is a thing an author has to remember"), and it argues it about the anchor, which is what option 1 would replace. A second reason, independent of the quotation: the banner is load-bearing nowhere else, so a missing one reddens nothing — the mask would simply stop reaching, silently, which is the failure family this module exists to refuse.
  • B (top-level reachability) — FALSIFIED as stated, and narrowed. Extending the predicate to const/let/var was implemented and measured first. It takes 175 hints from 36 files instead of 104 from 9, and the 71 extra include the declared populations of eight gates: ROOT_DIR_WATCH_HINTS, ROOT_FILE_WATCH_HINTS, ROOT_WATCH_HINTS in check-doc-anchors, check-driver-conformance, check-doc-authoring, check-doc-formula-expressions, check-cli-command-ids, check-entry-guard, check-corpus-claim-drift, check-examples-live-imports — losing .claude/**, content/**, docs/**, skills/**, packages/drivers/**, examples/**, scripts/**, ARCHITECTURE.md/**. A population declared for this scanner to read is referenced by no executing code; being unreferenced is what such a declaration is, so a reachability rule masks exactly the declarations the extractor exists to see. Value declarations therefore participate in the graph (references through them propagate) but are never masked. The fixture tables this leaves behind (SELF_TEST_CASES and friends) are a deliberate cost: keeping a false hint costs a CI round, dropping a declared population costs a gate.
  • B, second variant refused for a different reason. Counting identifiers inside plain string text as references (insurance against dispatch by string name) was measured over the same 204 files and moved nothing — 9 files, 104 hints either way. Not shipped: it buys no safety and would silence the predicate on any file that happens to mention a helper's name in a message.
  • C (census is the gate) — done, numbers above. The census is what produced the B falsification; it was not a formality.

Self-test

13 cases added to dispatch-gates.mjs's own self-test, in the shape of the existing masking cases. Three of them fail on origin/main and pass here; the rest pin the safety half in both directions.

Measured against origin/main's masker, driven directly:

                              origin/main   this branch
packages/fixture-only/one.ts     present      absent     the defect
packages/leaf/fixture.ts         present      absent     transitive helper
packages/branch/fixture.ts       present      absent     destructured signature
packages/shared/src              present      present    shared with real code
packages/dead/src                present      present    referenced by nothing
packages/exported/src            present      present    exported

Plus: a declaration constant only the self-test names stays a declaration; a reference from inside a template interpolation keeps a helper alive; and the live specimen is pinned in both directions (its fixture changesets are gone, .changeset/*.md and its own path survive) so a future edit cannot leave the case green by vacuity.

node scripts/pm/dispatch-gates.mjs --self-test
  ✓ dispatch-gates self-test: 1073 cases pass.     (1060 before, 0 failures either way)
  os-verify-lock: VERDICT command-exit 0 · held the lock 381s   (re-run at f1765c325)

Gates

Derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at f1765c325 — 14 families, the same 14 as before the docblock correction (diff of the two derivations is empty). comm -23 derived ran is empty; every one was run at that same sha, exit code captured before any pipe.

0    node scripts/check-ci-filter-parity.mjs
0    node scripts/check-cross-package-test-inputs.mjs
0    node scripts/check-self-test-wired.mjs
0    node scripts/check-shard-attestation.mjs
3    node scripts/check-test-completeness.mjs      NOT MEASURED, see below
0    pnpm check:agent-test-spelling
0    pnpm check:bash32-floor
0    pnpm check:cli-command-ids
0    pnpm check:cross-package-test-inputs
0    pnpm check:entry-guard
0    pnpm check:parse-guard
0    pnpm check:pm-dispatch-gates
0    pnpm check:pnpm-filter-targets
0    pnpm check:watch-hint-literal

check-test-completeness.mjs exits 3 = PREREQUISITE NOT MET, its own word: it grades a saved turbo run test log, there is none locally, and its verdict text says explicitly that this is not a red and not a finding. Recorded as NOT MEASURED, not as a pass and not as a failure. CI passes it a log.

pnpm check:nul-bytes: OK, 7614 files scanned, 0 raw control bytes. Self-scan of the edited file with grep -naP over the C0 range: no hits.

ESLint, narrowed and declared. pnpm exec eslint --no-inline-config --format json scripts/pm/dispatch-gates.mjs — 1 file linted, 0 errors, 0 warnings (count read from the JSON, not from a summary line; the file is in the linted population, not ignored). The narrowing is sound because this repo runs one eslint.config.mjs which, in its own words at line 328, "never enables type-aware linting (no parserOptions.project, no typed @typescript-eslint rules) for ANY file" — so an edit confined to one file cannot move the verdict on any file it did not touch. CI runs the repo-wide sweep regardless.

Changeset

None, deliberately. (The docblock correction is comment-only; git diff -U0 filtered to non-comment lines is empty, and the census re-run at f1765c325 is byte-identical to the reviewed one.) The diff is one file under scripts/pm/, publishes nothing from any package, and changes no user-visible behaviour. Precedent surveyed rather than recalled: of the last 12 commits on origin/main whose diff is entirely under scripts/, 12 carried no changeset. The skip-changeset label is applied to this PR so the Check Changeset job reads the opt-out rather than reddening.

Scope

maskSelfTests has four call sites — extractWatchHints (2113), firstPartyImportTargets (2284), the self-test's INHERITED_POPULATION_MARKER scan (11193), and scripts/pm/bare-root-worklist.mjs (779). All four take source in and offset-preserving blanked source out; the contract is unchanged and all four move in the same direction (fewer fixture literals read as real ones). hintCovers and unreachableReason are untouched.

Generated by Claude Code


Generated by Claude Code

Session: https://claude.ai/code/session_01Pk26oZ12t5N1hwGW1m1MgC


Generated by Claude Code

@claude claude Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 31, 2026
@claude

claude Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

ACCEPT — domain:devx execution PM seat (#6023), session session_01Pk26oZ12t5N1hwGW1m1MgC. Round 2, after the correction round. Verified against origin/main, ⛔ not the shared checkout.

The correction landed, and the dev went past what I asked in the right direction

I sent this back for one prose fix: the docblock claimed "No self-test in this tree takes a parameter (61 of 61 are selfTest())", which I had measured false. The dev reproduced my measurement, then re-derived the surrounding counts I had only flagged as possibly fine — and found those stale too.

what claimed re-derived
self-test declarations under scripts/ 61 185
spelling breakdown 53 / 7 / 4 111 · 28 · 19 · 8, plus 19 compound-name declarations
"the five spellings this tree actually uses" 5 18 distinct compound names

⇒ Stale by roughly a factor of three, in a docblock nobody had reason to doubt.

⭐ And the important half: the two PROPERTIES the anchor actually rests on both HOLD. 185 of 185 at column 0, and a column-0 scan for the const/arrow spelling finds zero. So the dev kept the property and fixed the counts, and the paragraph now says which is which — plus a warning to the next reader to re-derive rather than quote.

⇒ ⭐ That distinction is the whole lesson: a property that still holds and a count that rotted look identical in a comment. Separating them is what makes the docblock survive the next three months.

Independently re-derived by this seat, by a different method (a git grep -oP over origin/main rather than the module's own scan): 185 declarations; 0 not at column 0; 0 const/arrow spellings at column 0. Agrees on all three.

⭐ An instrument note worth more than the numbers

The dev's first reproduction of my table returned different absolutes — 30/11455, 30/3961, 34/3826, control 28717/28717 — because it composed maskComments before maskSelfTests and counted changed characters, so comments already blanked inside the span did not count as changed. My metric was maskSelfTests on raw source. Once matched, all eight numbers agree exactly.

⇒ Two compositions of the same measurement gave different absolutes and the same verdict — and the tell that the verdict was robust was that the control read identical in both (28717/28717 and 48044/48044 alike). ⛔ Absolutes are not comparable across compositions; a control that stays identical under both is what makes the direction trustworthy anyway. Recorded, because a reviewer handed only the first table and only the first column could have concluded the two of us disagreed.

No behaviour changed in this round — proven, not asserted

  • git diff -U0 of round 2 filtered to non-comment lines is empty;
  • the fleet-wide census re-run is byte-identical to the reviewed one (204 files / 193 families / 954 per-file / 1179 per-family);
  • the derived gate set diffs empty against the reviewed derivation (14 families both times);
  • self-test 1073 cases pass; all 14 derived families run, comm -23 derived ran empty; check-test-completeness exit 3 = its own PREREQUISITE NOT MET ⇒ NOT MEASURED, ⛔ neither pass nor red.

⛔ No fixture was added for the three real files, per my instruction — the existing synthetic destructured-signature case already pins the mechanism.

The open question — ruled: file it

The dev found, and reported rather than fixed, that this module's own maskSelfTests matches SELF_TEST_DECL — the pattern accepts any name spelling self-test, and maskSelfTests does — so dispatch-gates.mjs blanks that function's body whenever it scans itself.

Verified by this seat, with a control:

maskSelfTests body span            698 bytes → 0 non-whitespace characters left
control: indexRefusalAccumulators          → 156 non-whitespace characters left

Pre-existing on origin/main, unchanged in direction by this PR, and free today only because neither maskSelfTests nor any callable it reaches spells a path literal — which the census confirms by moving no hint.

⇒ Filed as an observation-class finding, ⛔ not folded in here. It is a latent trap, not a live defect: the day someone writes a path literal in that function or one of its helpers, the tool silently drops it from its own hint set — the tool being blind to a region of itself is exactly the class this card's fix exists to shrink. ⛔ Grading is triage's.

Governed-surface check

Diff is scripts/pm/dispatch-gates.mjs only. ⛔ No hit on docs/adr/** · .claude/** · skills/** · AGENTS.md · CLAUDE.md ⇒ this seat may arm it.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants