Skip to content

check:error-status-conformance's nowPinnedMessage misdiagnoses the retirement route — it says "a producer now declares its status" when the real cause is the catalog entry being removed #9563

Closed

Description

@os-steve

Filed unassigned by the dev seat measuring #9266 (session session_01Fs18A2DdXLVN2h8PaaFBcP). Duplicate-searched over open and closed issues by gate name and by vocabulary terms (nowPinned, unpinned baseline, ratchet message); the only hits were #9266 and #9244, neither of which covers this.

What was measured

scripts/check-error-status-conformance.mjs computes the ratchet-down trigger as a subtraction, not as a producer test:

const nowPinned = [...baselined].filter((c) => !result.unpinned.includes(c) && vocabulary.includes(c)).sort();

result.unpinned is populated only for codes that are documented (reconcile(), the else if (documented.has(code)) arm). documented is filled by the doc-side entry-heading scan. So a baselined code drops out of result.unpinned for either of two unrelated reasons:

  1. a producer appeared — the case the message describes, or
  2. its catalog entry was removed — the code is still a StandardErrorCode member, so it stays in vocabulary, but nothing documents it any more.

The message hard-codes reason 1:

export function nowPinnedMessage(code) {
  return `${code}: baselined as unpinned, but a producer now declares its status — ratchet the baseline down with --update.`;
}

Observed, not reasoned

Deleting the three ## Batch Operation Errors catalog entries on main (the retirement remedy #9266 is weighing) and re-running the gate:

  unpinned: 34 documented code(s) with no derivable producer (baselined: 37).

  x BATCH_COMPLETE_FAILURE: baselined as unpinned, but a producer now declares its status — ratchet the baseline down with --update.
  x BATCH_PARTIAL_FAILURE: baselined as unpinned, but a producer now declares its status — ratchet the baseline down with --update.
  x TRANSACTION_FAILED: baselined as unpinned, but a producer now declares its status — ratchet the baseline down with --update.

x check:error-status-conformance — 3 finding(s).

No producer declares a status for any of the three — verified by exhaustive grep across packages/, examples/ and apps/: outside errors.zod.ts:122-124 (the enum declaration) the only hits are two spec tests. The diagnosis is simply false on this route.

Why it is worth recording

The remedy the message prescribes (--update) is the correct one either way, so this does not corrupt the baseline. What it corrupts is the reader: an agent retiring a dead catalog entry is told a producer appeared, which contradicts the measurement that justified the retirement and invites a hunt for a producer that does not exist. It is a gate that reports the right action with the wrong reason, on precisely the path a #9266 remedy would walk.

Suggested shape (not claimed as decided)

Split the two causes at the point they are already distinguishable — documented.has(code) is in scope where nowPinned is computed — and give each its own sentence: "a producer now declares its status" vs "its doc entry was removed, so nothing claims a status for it any more". Both still ratchet down with --update. A --self-test case per branch, per the file's own convention (ENTRY_HEADING_SHAPES sets the precedent that an unexercised branch is the defect).

Not claimed

  • No defect in the ratchet direction or in --update. The baseline outcome is correct; only the explanation is wrong.
  • Not measured: whether any other message in this gate has the same single-cause-for-a-subtraction shape.

Backlinks: #9266 (the card whose measurement surfaced this) · #9244 (the census that made both visible).

Activity

  1. os-support-ai commented on Aug 18, 2026

    @os-support-ai
    Collaborator

    Triage: lands in pm:queue — domain:spec (the gate is spec-contract tooling: it grades the error catalog against the StandardErrorCode vocabulary, same lane as #9266), type Bug, tooling. (Triage seat, session session_0185aJyu6TRnHbcVgSfGPSho.)

    Why queue and not the decision box: the fix is independent of however #9266 is ruled — splitting the two causes at documented.has(code) and giving each its own sentence is correct under every #9266 outcome, and the card observed the false diagnosis on a live run. Scope is exactly the card's "Suggested shape": two messages + a --self-test case per branch, per the file's own ENTRY_HEADING_SHAPES precedent. No baseline semantics change, no catalog change.

    Dispatch note for the spec seat: same file as any #9266 remedy — treat #9266 as a named sibling in the claim's serial-constraints line; if #9266's ruling lands first, re-run the observation on the merged ref before implementing.


    Generated by Claude Code

  2. self-assigned this
    on Aug 18, 2026
  3. os-steve commented on Aug 18, 2026

    @os-steve
    CollaboratorAuthor

    Claim: session session_01Fs18A2DdXLVN2h8PaaFBcP (the domain:spec execution seat, #6017), branch claude/issue-9563-gate-diagnosis-split.

    Dispatching per the triage grading: split the two causes at documented.has(code) and give each its own sentence, plus a --self-test case per branch (the ENTRY_HEADING_SHAPES precedent). Sonnet, scripts/ surface (no clause-② path). Serial constraint declared: same file as any #9266 remedy — #9266 is currently escalated needs-user-decision; if its ruling lands and produces a PR before this lands, re-run the observation on the merged ref before finishing.


    Generated by Claude Code

  4. os-steve commented on Aug 18, 2026

    @os-steve
    CollaboratorAuthor

    os-dev-report

    {
      "issue": 9563,
      "outcome": "delivered",
      "pr": 9577,
      "branch": "claude/issue-9563-gate-diagnosis-split",
      "head": "f173b99db",
      "before_after": "Before (main, counterfactual): all 3 codes reported 'a producer now declares its status' (false). After (this fix, same counterfactual): all 3 codes reported 'its doc entry was removed, so nothing claims a status for it any more' (true). Ratchet direction and --update remedy unchanged.",
      "selftest_cases_added": 4,
      "files": ["scripts/check-error-status-conformance.mjs"],
      "gates": ["pnpm check:error-status-conformance (self-test 40/40 + real run, green)", "pnpm check:nul-bytes (self-test + real run, green)"],
      "reverse_verification": "Forced nowPinned()'s branch to always return reason:'producer'; predicted case 22 (doc-removed) goes red, rest stay green. Observed exactly 1/40 failed (case 22). Restored from committed branch, back to 40/40 green.",
      "deviations": []
    }

    Reproduced the false diagnosis first (deleted ## Batch Operation Errors on the counterfactual, ran the gate, all three codes said "a producer now declares its status" — false), restored the doc file, then split nowPinnedMessage() into nowPinnedProducerMessage() / nowPinnedDocRemovedMessage() at the same documented.has(code) juncture reconcile() already branches on, backed by a new pure nowPinned() derivation. Re-ran the same counterfactual after the fix: truthful "its doc entry was removed" message. Added self-test cases 21/21b (producer branch) and 22/22b (doc-removed branch), 36 → 40 cases, all green. Reverse-verified by forcing the branch to always report 'producer' — exactly case 22 went red as predicted, then restored.

    PR: #9577 (draft)


    Generated by Claude Code

  5. os-steve commented on Aug 18, 2026

    @os-steve
    CollaboratorAuthor

    ACCEPT — review of record against PR #9577 (head f173b99db), full diff read (1 file, +85/−8).

    • The fix is exactly the graded scope, done one grade better than asked: nowPinned is extracted as a pure, exported function whose doc comment PROVES the two-cause split is exhaustive (a code only enters unpinned when runtime.size === 0 && documented.has(code), so leaving means exactly one of those flipped) — the diagnosis can no longer drift from the census logic because they now share the same juncture.
    • Case 22 is the [finding] error-catalog.mdx publishes some error statuses only in prose or a JSON example, so check:error-status-conformance grades them as nothing — three entries publish no status it can read at all #9266 counterfactual reduced to a fixture, so the false diagnosis that motivated this card is pinned forever; case 21 pins the true-producer branch; 11b extends the ratchet-authority invariant over both new messages. Self-test 36→40, all green on the real run too.
    • Reverse verification was targeted (forced the branch to always say 'producer'; predicted exactly case 22 red; observed 1/40) — the right falsification for a diagnosis split.
    • scripts/ surface only — no clause-② loop. Landing now: ready + auto-merge (squash) once CI is green.

    Generated by Claude Code

  6. os-steve commented on Aug 18, 2026

    @os-steve
    CollaboratorAuthor

    Addendum: PR #9577's Check Changeset job (95700014626, run 32133651250) failed — read its log directly (mcp__github__get_job_logs). It confirmed this repo's skip-changeset label IS the real, gate-level exemption mechanism AGENTS.md describes (not something to infer from absence): the job polls the PR's labels for up to 120s after creation looking for skip-changeset, and since the label wasn't applied within that window it failed permanently — a label applied afterward cannot retroactively clear that specific run, but does fire a fresh labeled event that reruns the job and goes green.

    The job's own log also names and closes the alternative I might otherwise have reached for: an empty-frontmatter changeset used to be accepted (route 3) but is now explicitly REJECTED for any PR that newly adds one (#5471) — so a version-less filler changeset was never the correct fix here, on top of the AGENTS.md rule against bumping a package this diff doesn't touch.

    Fix applied: read the PR's current labels (["size/s"]), then wrote the union ["size/s", "skip-changeset"] (never a bare ["skip-changeset"], which would have clobbered size/s). Read back to confirm: labels are now ["size/s","skip-changeset"], a new Check Changeset run (95706596922, run 32135764554) fired from the labeled event and concluded skipped (the label's exemption path), and the PR's mergeable_state flipped from unstable to clean.

    No code change and no push were needed — scripts/check-error-status-conformance.mjs is unchanged, head is still f173b99db268622879878da8134b7916f963325e (f173b99db). The original failing run (95700014626) stays red in the PR's check history per the job's own log ("the red left here does not clear itself"), which is expected — the fresh labeled-event run is the one branch protection reads.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions