Skip to content

fix(verify): read a refused create's whole answer before calling it a hard failure - #24516

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-24498-verify-validation-cut
Oct 19, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-24498-verify-validation-cut

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #24498

Clause-②: no

Basis: the CRUD stage reads the same answer correctly. No accept set, export or published shape moves.

What changed

packages/verify/src/verify.ts, runCrudVerification, the refused-create branch:

  • The body is read once, in full. The verdict comes from a new private helper, isValidationAnswer(text).
    • When the body is JSON with a string top-level code, a 400 is needs-fixture exactly when that code is VALIDATION_FAILED.
    • When the body is not JSON, or has no string code, the stage's original pattern (VALIDATION_FAILED|Validation failed, case-insensitive) is matched against the full text.
  • The stored detail keeps its 280-character cut (text.slice(0, 280)), taken after the verdict.
  • The rule is unchanged: a 400 validation answer is needs-fixture, and anything else is create-failed. No other status value, exit-code rule or verify stage is touched.

The shape was measured, not guessed. A real in-process stack (bootStack, this branch, after the dependency closure was built) answered a create refused by a script rule with 400 and a 755-character body whose keys are ["error","code","fields","object"]. The "code":"VALIDATION_FAILED" key starts at character 324. That is the flat envelope packages/types/src/data-error-classification.ts builds for the validation arm (error = the app's message, then code, fields, object). It is also the top-level code that the sibling stages already read off this same door (reference-shape.ts, delete-behavior.ts, unique-scope.ts). The test fixture uses that measured key order and fields entry shape.

Before and after: the stored row for a long validation answer, measured

The probe is a throwaway script, not committed. It boots one real stack with one object whose script rule (CEL true, events insert and update) refuses every create with a three-sentence message, like hotcrm's forecast rule. It then runs runCrudVerification over that app. "Before" imports verify.ts exactly as it is at base 6b42b6bc8. "After" imports it at 3024fa2b7.

run verify.ts status code detail length summary
before base 6b42b6bc8 create-failed 400 280 createFailed: 1, needsFixture: 0
after 3024fa2b7 needs-fixture 400 280 createFailed: 0, needsFixture: 1

The stored detail is byte-identical in both rows. It ends ... reconcile the three amounts, th, before the code key at character 324. Only the verdict moves.

The optional hotcrm re-run was not done. The hotcrm checkout has no node_modules. Running os verify there with this branch's build means installing its 18.0.0-next.3 dependency tree and building and wiring @objectstack/cli from this branch, which is more than linking the built package. The in-process probe above drives the same door and the same stage.

Ablation

The ablation restores today's cut-before-classify order with one mutation in verify.ts. It goes through node scripts/ablation-replace.mjs (WRAP mode, its own exit, INT and TERM trap, plus a shell trap that runs git checkout HEAD -- on the absolute path):

  • Before the mutation, the line read const text = await created.text();. The mutation made it const text = (await created.text()).slice(0, 280);.
  • The anchor was hit 1 time and dropped to 0. The replacement count went from 0 to 1. The blob moved from d8fe2c05c833 to d9528c3cad95.
  • With the mutation, vitest run src/verify.create-refusal-verdict.test.ts gave 1 failed, 4 passed. The one red is the long-message case, expected status "needs-fixture", received "create-failed". All four controls stayed green.
  • After the restore, the blob read d8fe2c05c833, which equals HEAD. git diff HEAD was empty and git status --porcelain was empty.

The test imports ./verify.js from source (a relative import, so there is no dist/ resolution), so no rebuild leg was owed.

Controls

The new test file is packages/verify/src/verify.create-refusal-verdict.test.ts. It uses a stack double (apiAs only, the same double pattern reference-shape.test.ts uses) and boots nothing. It has 5 cases, and all pass on HEAD:

  1. A 400 VALIDATION_FAILED whose message runs past 280 characters is needs-fixture, and detail equals the first 280 characters of the body. The fixture asserts its own precondition: the code key starts after character 280.
  2. Control: a short 400 VALIDATION_FAILED is needs-fixture.
  3. Control: a 500 INTERNAL_ERROR is create-failed.
  4. Control: a 400 INVALID_FIELD is create-failed.
  5. Control: a 400 non-JSON Validation failed: ... body is needs-fixture. This exercises the full-text fallback.

Local verification (HEAD 3bcc4ae59)

  • pnpm --filter @objectstack/verify exec vitest run --maxWorkers=2 (the whole package suite, under the shared lock) gave 44 test files and 376 tests passed, VERDICT command-exit 0.
  • pnpm --filter @objectstack/verify typecheck exited 0. It runs tsc --noEmit and then check:test-typecheck, which reported "0 file(s) / 0 error(s)". tsc -p tsconfig.test.json --listFiles lists the new test file (1 hit). For comparison it lists 44 verify test files and 100 packages/core/src files.
  • eslint (--no-inline-config) over verify.ts and the new test exited 0, with 0 errors and 0 warnings. The JSON output confirms both files were linted.
  • node scripts/pm/dispatch-gates.mjs --commands derived 63 commands. All 63 were run with each exit code captured before any pipe, and all exited 0. pnpm check:dual-build-cjs-loads first answered PREREQUISITE NOT MET (exit 3, unbuilt packages elsewhere in the tree). It was re-run after a full turbo run build (75 of 76 tasks cached) and exited 0.
  • The --ran reconciliation read: 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN (a derived zero, all 63 with recorded exit codes).
  • pnpm lint (eslint . --no-inline-config), the full union, was run at 3bcc4ae59 (the final commit) and exited 0 with no findings printed.

Acceptance notes

  • A JSON 400 whose top-level code is something other than VALIDATION_FAILED, but whose prose contains "Validation failed", now reads create-failed. The old pattern over the cut made it needs-fixture. On the /data create door, the only producer that writes that phrase into the flat envelope is the VALIDATION_FAILED arm itself (data-error-classification.ts). The other producers of the phrase are other doors: plugin-auth's adapter, the import runner, and the CLI's own printers. So no answer the door gives today changes its verdict except the long-message case.
  • There are no out-of-scope findings. No other verify stage cuts a body before it classifies it: the sibling stages parse the full text, and their clip is applied to stored detail only.

Generated by Claude Code

…tored cut

The CRUD stage cut a refused create's body to 280 characters and then
tested that cut for the validation code. The /data door's validation
envelope puts the app-authored message before `code`, so a long message
pushed `"code":"VALIDATION_FAILED"` past the cut and a fixture gap was
reported as a hard `create-failed`. The verdict now reads the whole body
(the envelope's top-level `code` when the body is JSON, the original
pattern over the full text otherwise); the 280-character cut stays on the
stored `detail` only.

Claude-Session: https://claude.ai/code/session_01SYEJqDL3NXxrJRBVqaMu9T
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation tests tooling labels Oct 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/verify, touching 3 documentable anchor(s).

24 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 913fd6df8a2b3ec82612bf35dc40f4e062c3077f.

⛔ 5 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 913fd6df8a2b3ec82612bf35dc40f4e062c3077f → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 3630156e255cf4aa8f74f74b8612d63f0613c064 — the merge of head 3bcc4ae59269cbf7ebadc2ca75a008967843a0e1 into base 913fd6df8a2b3ec82612bf35dc40f4e062c3077f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 3630156e255cf4aa8f74f74b8612d63f0613c064 && git checkout 3630156e255cf4aa8f74f74b8612d63f0613c064
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 913fd6df8a2b3ec82612bf35dc40f4e062c3077f 3bcc4ae59269cbf7ebadc2ca75a008967843a0e1 && git checkout -B drift-repro 913fd6df8a2b3ec82612bf35dc40f4e062c3077f && git merge --no-ff 3bcc4ae59269cbf7ebadc2ca75a008967843a0e1

node scripts/docs-audit/affected-docs.mjs --json 913fd6df8a2b3ec82612bf35dc40f4e062c3077f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 913fd6df8a2b3ec82612bf35dc40f4e062c3077f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

席位核对 — ACCEPT

domain:cli seat 1 (seat post #6024) · zhuangjianguo · session session_01SYEJqDL3NXxrJRBVqaMu9T · 2026-10-19T10:18Z · on head 3bcc4ae59 · #24498 (p2, claim 6264271985, report 6264919406). This is the seat's own review at the default tier, read against the diff on GitHub, not the report.

  1. What moved, read on the compare 6b42b6bc8...3bcc4ae59:
    • runCrudVerification reads a refused create's body once, in full, and classifies through a private isValidationAnswer(text).
    • For a JSON body with a string top-level code, the verdict is code === 'VALIDATION_FAILED'. Anything else is matched against the full text with the stage's original pattern.
    • The stored detail keeps text.slice(0, 280). Statuses and the exit rule are unchanged.
  2. The one narrowing, checked by the seat. A JSON body whose code is set and is not VALIDATION_FAILED, but whose text says "Validation failed", is now create-failed (before, a match in the first 280 characters made it needs-fixture).
    • packages/types/src/data-error-classification.ts maps every validation refusal to code: 'VALIDATION_FAILED': the ValidationError arm (about :1183–:1188, with the message 'Validation failed' as its fallback) and the required-field arm (about :1905–:1913).
    • The other 400 codes (INVALID_FIELD and the rest) carry no such text.
    • So the envelope's code is the contract, and the text pattern stays as the fallback for a body that carries none. No app's validation refusal changes verdict except the long-message case this card is about.
  3. Shape:
  4. Tests and reverse verification:
    • verify.create-refusal-verdict.test.ts: 5/5 against a stack double. The long-message case asserts the precondition that the code key sits past character 280, and that detail equals the 280-character cut.
    • The controls cover a short 400 VALIDATION_FAILED, a 5xx, a 400 with another code, and a non-JSON 400.
    • Ablation, restoring the cut-before-classify order: only the long-message case goes red. The restore was proven by blob.
    • Live probe on a real bootStack with a three-sentence rule message: create-failed at base, needs-fixture at head, with the same 280-character detail.
    • The whole package passes (44 files, 376 tests), and typecheck exited 0.
    • The hotcrm re-run was skipped, with its reason stated: it needs hotcrm's install plus a wired CLI from this branch.
  5. Gates:
    • dispatch-gates: 63 derived, 63 run, 0 NOT-MEASURED. One family was first NOT MEASURED for unbuilt dependencies and passed once they were built.
    • The full-union pnpm lint and changed-file eslint exited 0.
    • Landing waits on every check of this head.
  6. Findings: none.
  7. Landing: ready plus auto-merge through the queue once every check of 3bcc4ae59 is green or a rostered skip. The card closes on the merge.

Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 19, 2026 10:38
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 19, 2026 10:38
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit f7eb860 Oct 19, 2026
41 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-24498-verify-validation-cut branch October 19, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

2 participants