Repository navigation
fix(rest): a NOT NULL refusal's hint leads with the remedy for a declared column and names schema drift only as a condition - #21001
Conversation
…mn that requires a value, and names schema drift only as a condition A field declaring storage.notNull without required is a column the database keeps NOT NULL on purpose (ADR-0113 D1 / D2). A write with no value for it is refused by the database and answered 400 VALIDATION_FAILED with a required finding, which is right; the hint said the schema had drifted and sent the author to os migrate, which changes nothing for that column. The hint now says the column requires a value (provide it, or declare the field required) and names drift in a second sentence, conditional on the object declaring neither required nor storage.notNull. The branch comment gets the same correction. code, status, fields and the sentence are unchanged, no registry is read, and the import row is untouched. Claude-Session: https://claude.ai/code/session_01VvcEokUG1tvVxkceYfR5XB Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 15 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 928a24d445a78dbd5765cef7490bc86c00490d97 && git checkout 928a24d445a78dbd5765cef7490bc86c00490d97
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 8d329f02eeb571170bba3b275df3c2fc3cb749b3 592069136c016aec922b78fd9b977c80f870b062 && git checkout -B drift-repro 8d329f02eeb571170bba3b275df3c2fc3cb749b3 && git merge --no-ff 592069136c016aec922b78fd9b977c80f870b062
node scripts/docs-audit/affected-docs.mjs --json 8d329f02eeb571170bba3b275df3c2fc3cb749b3 |
Contract reviewServed-tier: Inputs read: card #20963 body and its three comments (governing: triage's direction ① Derived judgments(a) The not-null branch of
(b) The pin
(c) Confined to the claim's surface. Three paths. (d) The changeset ② Semver level
③ Boundary flagsDev
Dev
Serial interaction with #20919: the cross-lane note File list: three paths, none a governed surface ( Check-runs on Implemented-by: VERDICT: PASS Generated by Claude Code |
…s a mapping's connectorSource through the import runner, moved beside bulkWrite (objectstack-ai#21084) Fixes objectstack-ai#20919 Clause-②: yes (widening) Stage ② of ruling A on objectstack-ai#20281: the **connector sync executor**. `service-automation` pulls a `mapping` whose `connectorSource` names a declared `rest` / `openapi` connector, and writes the records through the import runner's upsert-by-match-key. Built to the seat's stage-② direction (`5922160386` on objectstack-ai#20281) and the amended claim surface (`5922189483`). ## What changed **1. The move (destination A, 0 new `package.json` edges).** `service-automation` → `rest` would invert a layer, so the runner's shared pieces moved down. ⛔ None of them is a copy. - To `@objectstack/core` (`src/utils/`, beside `bulkWrite`): `import-runner.ts` and `import-coerce.ts` (whole files), the mapping apply (`applyMappingToRows`, `refuseUnknownMappingTargets`, `MappingArtifactLike`, `MappingFailure`) as `import-mapping.ts`, and the field meta (`ExportFieldMeta`, `buildFieldMetaMap`) as `import-field-meta.ts`. Their suites moved with them: 6 `import-runner-*` suites, `import-coerce`, `import-mapping`. - To `@objectstack/types`: the classification half of `rest/src/error-response.ts` (`CLIENT_MESSAGE_MAX` through `classifyDataError`, 1,892 lines) as `data-error-classification.ts`. `rest` keeps the emitters. - `rest` re-exports every moved name: from `export-format.ts`, `import-mapping.ts`, `error-response.ts` and the package index. Its production callers are unchanged: `rest-server.ts`'s two import routes, and `plugin-auth`'s `admin-import-users.ts` through the index. The one exception is an import line in `rest-server.ts` (`runImport` now comes from `@objectstack/core`). - Two suites stayed in `rest` because they need `objectql` / `metadata-protocol`, which `core` cannot depend on: `import-runner-historical-readonly-insert`, `import-runner-unique-violation-row`. They now import from the new home. **2. The executor:** `packages/services/service-automation/src/connector-pull.ts`. It is exposed as `pullConnectorSource(deps, opts)` and as `AutomationServicePlugin.pullConnectorSource({ mapping, context })`. - It reads the mapping through the protocol's `getMetaItem({ type: 'mapping', name })`, as the import door does. - It resolves `connectorSource.connector` to a **declared** instance. The plugin now records each materialized instance's provider, because the registered def carries none. Only `rest` / `openapi` is accepted. - It makes **one** action call per pull. `watermark.param` goes into the call's `query`. The records are read at `recordsPath` (default `body`). - **W1 watermark.** The starting point is the highest value stored in the target field that a transform-`none` `fieldMapping` entry copies `watermark.field` onto. It is one read, under the caller's context. Nothing else stores it. - It writes through the moved `runImport`. Every knob `connectorSource` does not declare takes the door's default: `runAutomations` true, `trimWhitespace` true, the rest false. - Nothing schedules a pull. That is stage ③ (a `job`). **3. The texts the executor makes true** (`domain:spec`, the amendment's list): - the `connectorSource` and `watermark` describes; - the `mapping.json` ledger rows: `planned` → `live`, citing `connector-pull.ts#pullConnectorSource`. `authorWarn` stays, with the hint reworded to "pulled when a job drives it; nothing schedules it until stage ③"; - the `syncConfig` prescription and its pin; - the D3 entry (it drops "validated at authoring"); - `SYNC_ARCHITECTURE.md`: how a pull runs, the one-response limit, and the two stale Best Practices bullets rewritten; - the `packages/lint` group-C comments and test title. The migrations registry region and `content/docs/references/**` were regenerated with `check:generated --fix` and `gen:schema` / `gen:docs`. ⛔ No new key, cadence key, credential or paging convention. ## The executor's refusals Every refusal is a typed `ConnectorPullError` (`code`, `status`, `reason`). It is raised **before anything is written**, and each pin asserts that no write reached the protocol. The codes are standard-catalog members or the code the import door already answers with, so no new ledger code was minted. | Condition | `code` | `status` | `reason` | |---|---|---|---| | action answered `ok: false` | `EXTERNAL_SERVICE_ERROR` | 502 | `upstream_not_ok` | | non-array at `recordsPath` | `INTEGRATION_ERROR` | 502 | `records_not_array` | | a record that is not an object | `INTEGRATION_ERROR` | 502 | `record_not_object` | | plugin-origin connector (not a `connectors[]` entry) | `VALIDATION_ERROR` | 400 | `connector_plugin_origin` | | unmapped `watermark.field` (none, or only through a transform) | `VALIDATION_ERROR` | 400 | `watermark_field_unmapped` | | `update` / `upsert` with an empty `upsertKey` | `VALIDATION_ERROR` | 400 | `upsert_key_empty` | | unknown mapping | `MAPPING_NOT_FOUND` | 404 | `mapping_not_found` | | mapping without `connectorSource` | `VALIDATION_ERROR` | 400 | `no_connector_source` | | connector not registered | `VALIDATION_ERROR` | 400 | `connector_not_registered` | | degraded instance | `SERVICE_UNAVAILABLE` | 503 | `connector_degraded` | | provider not `rest` / `openapi` | `VALIDATION_ERROR` | 400 | `connector_provider_unsupported` | | action the connector does not declare | `VALIDATION_ERROR` | 400 | `connector_action_unknown` | | `javascript` transform | `UNSUPPORTED_TRANSFORM` | 400 | `unsupported_transform` | | `connectorSource.input` / `input.query` not an object (watermarked pull) | `VALIDATION_ERROR` | 400 | `input_query_not_object` | | target that is no field of the object | `INVALID_FIELD` (the door's) | 400 | `mapping_target_refused` | The first six are the order's mandated refusals. ## Pins - `connector-pull.test.ts` (18 cases). Every refusal above, plus the watermark read: the target field, highest first, `limit: 1`, merged into the declared `input.query`. A first incremental pull sends no starting point. - `connector-pull.integration.test.ts`: **end to end through a real `rest` connector**. - A declared `connectors[]` entry is materialized by the real `ConnectorRestPlugin` provider. It reads a local `node:http` fixture server over a real socket, and writes into a real ObjectQL engine on SQLite. - Pull 1 sends no `since` and creates 2 rows. - Pull 2 sends `since=2026-01-02T00:00:00.000Z`, the max `synced_at` **read from the target**. It **updates** `c2` by `upsertKey` and creates `c3`. - It imports `connector-rest`'s source by relative path. That package already dev-depends on `service-automation`, so a manifest edge back would close a cycle. The read is declared in `scripts/cross-package-test-inputs.mjs` and `turbo.json`, and `service-automation`'s type-check `rootDir` is widened to admit it (tsup emit unaffected; `check-dts-emitted` 2/2). - Red first: `4f4260ee` committed the pins alone. The run against that state failed both files: `Cannot find module './connector-pull.js'` and `automation.pullConnectorSource is not a function`. `4b301797` then added the code. The executor file's hash was identical before and after that red run (`26a83b0d4a20fdab`). - The moved suites are green at their new homes: `core` 69 files / 2044 tests, `rest` 241 / 4576. ## Verification ### Move proof (a move is a move) An extraction script compared every moved range at base `e161ad35` with its destination: ``` IDENTICAL packages/rest/src/export-format.ts:35-65 (31 lines) -> packages/core/src/utils/import-field-meta.ts IDENTICAL packages/rest/src/export-format.ts:139-182 (44 lines) -> packages/core/src/utils/import-field-meta.ts IDENTICAL packages/rest/src/import-mapping.ts:37-55 (19 lines) -> packages/core/src/utils/import-mapping.ts IDENTICAL packages/rest/src/import-mapping.ts:117-328 (212 lines) -> packages/core/src/utils/import-mapping.ts IDENTICAL packages/rest/src/error-response.ts:66-1957 (1892 lines) -> packages/types/src/data-error-classification.ts WHOLE-FILE packages/rest/src/import-runner.ts -> packages/core/src/utils/import-runner.ts: 7 changed line(s), all import lines WHOLE-FILE packages/rest/src/import-coerce.ts -> packages/core/src/utils/import-coerce.ts: 9 changed line(s), all import lines ``` `git diff -M --color-moved=zebra` over the split files, counting `+`/`-` lines painted as moved against those not: ``` packages/types/src/data-error-classification.ts: moved+=1896 non-moved+=49 (header, import lines, trailing export list) packages/rest/src/error-response.ts: moved-=1896 non-moved+=28 non-moved-=5 (header note, import/re-export lines) packages/core/src/utils/import-mapping.ts: moved+=241 non-moved+=14 (header, import lines) packages/rest/src/import-mapping.ts: moved-=241 non-moved+=11 (import/re-export lines) packages/core/src/utils/import-field-meta.ts: moved+=75 non-moved+=13 (header) packages/rest/src/export-format.ts: moved-=75 non-moved+=4 non-moved-=1 (import/re-export lines) ``` The eight classification helpers the REST emitters compose are exported through a **trailing** `export { … }` list, so no declaration line was edited. Never a copy: no declaration of `mapDataError`, `classifyDataError`, `structuredCodeAnswer`, `declaredHttpStatus`, `applyMappingToRows`, `refuseUnknownMappingTargets`, `buildFieldMetaMap`, `runImport`, `coerceRow`, `ExportFieldMeta` or `MappingArtifactLike` is left in `packages/rest/src` (0 each). The control: the same grep finds each one at its new home. R2 held: the moved half has 0 code references to the emission half or to `log.js`. Its only mentions of the emitters are `{@link}` prose in docblocks. ### objectstack-ai#20963 (PR objectstack-ai#21001) It had landed (`e161ad35`) before the split, so the branch starts from it. The moved half carries objectstack-ai#21001's 18 added lines byte-equal (`18 of 18` found verbatim in `data-error-classification.ts`). There was no git conflict. ### Tests and typechecks Run on head `ec59b16a`, after merging `main`: - `types` 22 files / 685 tests; - `core` 69 / 2044; - `rest` 241 / 4576 (+135 skipped), `test:repo` 5 / 179; - `service-automation` 162 / 2021; - `spec` 587 / 17235; - `plugin-auth` import suites 2 / 50; - `lint` 118 / 5452; - `runtime` vocabulary suites 4 / 70. `typecheck` is green for `types`, `core`, `rest`, `service-automation`, `spec`, `lint`, `runtime` and `plugin-auth`. `spec check:generated`: all 15 artifacts up to date. `check:liveness`: green. ### Gates `dispatch-gates --ran` reconciles: 139 derived, 138 run, 1 NOT-MEASURED, 0 UNRUN, run on head `ec59b16a`. - **NOT MEASURED: `check:pm-dispatch-gates`.** Its non-self-test half outlives the ~10-minute foreground cap here (killed at 595s); its `--self-test` half exits 0. CI runs it. - **One red, not this PR's: `check:platform-checklist`.** The only finding is `areas/identity-auth.json` → `plugin-auth/src/auth-plugin.ts#twoFactor`, and it is red identically at base `e161ad35` and on `origin/main`. This PR's own checklist anchors (`classifyDataError`, the mapping `transform` switch) are repointed and resolve. - The 11 ⛔ roster families the derivation printed are all exit 0. ## Changed lines **6,032** (`additions + deletions`, rename detection on: 3,672 / 2,360, 64 files), over the 5,000 line. Per AGENTS.md this PR takes the **human-merge path** (an authorised approval, then the owning seat, or the maintainer's merge), not the queue alone. About 4,100 of those lines are relocation, and GitHub shows the whole-file moves as renames. ## Acceptance notes **Carriers, not touched here:** - **ADR-0114**'s consumer line names `@objectstack/rest`'s `import-coerce.ts` / `import-runner.ts`. That is `docs/adr` (Tier H), left for a docs follow-up. - `packages/objectql` comments cite `packages/rest/src/import-runner.ts` (`index.ts`, `summary-errors.ts`, and two `plugin-audit-created-*` test comments). They are outside the surface. - Further comment-only path citations of the moved files live in `driver-turso`, `metadata-protocol`, `plugin-security`, `spec` (`conversions/registry.ts`, `api/export.zod.ts`, `data/mapping.zod.ts`'s older docblocks), `scripts/check-dispatcher-error-vocabulary.mjs`, `check-error-status-conformance.mjs` and `check-durability-degradation-log-level.mjs`. They are prose only, and every gate is green. - `.changeset/20281-connector-sync-moved-to-mapping.md` (stage ①, unreleased) still says the binding is "declared, not yet executed". This PR's spec changeset supersedes it; whoever compiles the release notes should read the two together. **Surface extensions, each a mechanical consequence of the move or of the pin:** - `turbo.json` and `scripts/cross-package-test-inputs.mjs`: `rest`'s source-reading pins read the moved runner and classification, and `service-automation`'s integration pin reads `connector-rest` source; - `packages/rest/vitest.repo-tests.json`; - `packages/services/service-automation/tsconfig.json` (`rootDir`); - `docs/qa/platform-checklist/areas/{attachments-storage,records-forms}.json` anchors; - `packages/spec/liveness/state-counts/mapping.md` (generated). **Choices the direction left open, settled here:** - The refusal codes come from the standard catalog, so the ADR-0112 ledger is not edited. `reason` is the discriminator. - The watermark only reads a transform-`none` `fieldMapping` entry: a transformed target does not hold the pulled value. **Not measured:** - a real third-party `openapi` target (the pin uses `rest`); - the paging behaviour of any real upstream (no measured target exists; the one-response limit is documented); - `spec test:repo` was narrowed to the 7 repo suites that read this diff's files. The full project outlives the foreground cap here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- _Generated by [Claude Code](https://claude.ai/code/session_012Kijp4k6e9fvgq63BLGsxY)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20963
Clause-②: no
The
hinton a driver's NOT NULL refusal no longer asserts that the schema has drifted. It first says the column requires a value (provide it, or declare the fieldrequired), then names drift in one sentence that is conditional on the object declaring neitherrequirednorstorage: { notNull: true }. Prose only, in the one branch of the driver-string classifier inpackages/rest/src/error-response.ts; the branch's comment gets the same correction (ADR-0113 D1 / D2). The status,code,error,fieldsandobjectare unchanged, no key is added, no registry is read, and the import row is untouched (PR #20956 pins its key set).Repro (
hint, same answer on both doors)Object
proj_notnull_hint_20963declaresmust: { type: 'text', storage: { notNull: true } }with norequired.POST /api/v1/data/:objectwithmustomitted, andPATCH /api/v1/data/:object/:idwritingmust: null, over a real ObjectQL + SQLite:memory:(the pin's harness). Both answer400 VALIDATION_FAILED,error: "must is required",fields: [{ field: 'must', code: 'required', message: 'must is required' }],object.origin/main05be352, the pin run red on it): "If 'must' is optional in your object metadata, the database column is still NOT NULL — the physical schema has drifted from metadata. Run 'os migrate' to reconcile (or reset the dev database)."requiredin the object metadata. If the object declares neitherrequirednorstorage: { notNull: true }for 'must', the physical schema has drifted from metadata instead: run 'os migrate' to reconcile (or reset the dev database)."What is pinned
packages/rest/src/data-not-null-hint-20963.test.ts, three tests, by kind and not by wording:required;code,error, therequiredfinding andobjectas before, and the body's key set is exactlycode, error, fields, hint, object;hint: its first sentence says the column requires a value, tells the caller to provide it or declarerequired, and names no drift; drift appears in exactly one later sentence, which begins with a condition namingstorageandnotNull; the update door answers the create door's hint byte for byte.Not pinned: telling the two cases apart. That needs a registry read, a wider change than this card asks.
Verification (head 5920691)
origin/mainbefore the edit: 2 failed, 1 passed (the fixture test). With the fix: 3 passed.scripts/ablation-replace.mjsput the old hint text back (anchor hit 1 time, blob 0b1a39350 to 3197ecfe1; the mutated file resolves by relative source import from the test, so no rebuild applies) and the pin went red: 2 failed, 1 passed (the control and fixture held; the hint tests failed). The restore provedblob == HEADand an emptygit diff HEAD. Direction: turned red, the usual one.pnpm --filter @objectstack/rest typecheckexit 0 (tsc --noEmit, andcheck:test-typecheckovertsconfig.test.json: 0 files, 0 errors; the new test is in that program).vitest run --project localon@objectstack/rest: 250 files passed, 4964 tests passed, 114 skipped. The ten files that read this mapper, this file included, were also run alone: 10 passed, 567 tests.dispatch-gates --commandsderived 61 commands at this head; all 61 were run with exit codes recorded and--ranreconciled: 61 derived, 59 run, 2 NOT MEASURED, 0 unrun. The two NOT MEASURED exited 3, PREREQUISITE NOT MET:check:dual-build-cjs-loadsandcheck:type-check-debtread every package's built output and need the full-repo build, which was not run on the shared box;Build CoreandLint & Repo Gatescover them in CI. The diff adds no package, no export and no ledger entry, and the package's own test-layer typecheck holds 0 debt.pnpm lint(eslint over the whole repo,--no-inline-config) exit 0 at this head, no narrowing.Acceptance notes
Noted, not filed (no defect, polish only):
packages/rest/src/rest.test.ts(the test "maps SQLite NOT NULL constraint → 400 VALIDATION_FAILED with required field", about:2514) keeps a comment that says reaching this branch means the metadata did not require the field, so it is physical-schema drift. Its assertions still hold (the hint keepsos migrateand "drifted from metadata" in the conditional sentence), so only the comment restates the corrected reading. It sits outside this claim's file surface, so it is untouched.packages/rest/src/error-response.ts:176lists "not-null drift" as a label for the branch in a docblock about a different branch. A name, not a claim about the hint.origin/main7fa67da (onlyscripts/pm/fleet-write/moved) with--force-with-lease, before any PR existed, so the dispatch-gates families were derived from a current tree.Generated by Claude Code