Skip to content

fix(driver-sql, driver-turso): SQLite $contains / $notContains / $icontains / $endsWith read the whole stored value, not only up to its first U+0000 - #20038

Merged
objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-20024-sqlite-glob-stored-nul
Sep 24, 2026
Merged

objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-20024-sqlite-glob-stored-nul

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Part of #20024

Clause-②: no

What this changes

This takes item 1 of #20024, the stored value's cut. Item 2 ($like / $ilike) stays open on #20024 for a decision; see "Item 2" below.

On the SQLite faces, $contains, $notContains, $icontains and $endsWith compiled to GLOB for a comparand without U+0000. SQLite's glob() reads the stored value as a C string, only up to its first U+0000. So $contains: 'b' missed a value stored as 'a' + U+0000 + 'b', $endsWith: 'a' returned it, and $icontains: 'B' missed 'A' + U+0000 + 'B'. Nothing raised. #19999 (PR #20019, 9d81af714f) had already moved a comparand that HOLDS U+0000 onto length-aware constructs; this sends every contains / ends comparand there, which is option A of contract review 5821310673 (reconfirmed in 5822054158).

shape SQLite predicate, every comparand
contains ($contains, $notContains, $icontains) instr(col, ?) > 0
ends ($endsWith), non-empty comparand coalesce(substr(CAST(col AS BLOB), -length(CAST(? AS BLOB))), CAST(col AS BLOB)) = CAST(? AS BLOB)
ends, empty comparand instr(col, ?) > 0
starts ($startsWith), comparand without U+0000 col GLOB ?, unchanged byte for byte
starts, comparand holding U+0000 instr(col, ?) = 1, unchanged from #19999
  • The empty $endsWith comparand needed its own route. In the suffix construct -length('') is -0, which substr reads as "from the start", so the construct would answer false for 'abc'. Measured on all three engines: the suffix construct with '' answers 0 on 'abc', while instr('abc', '') and instr('', '') are both 1 and instr(NULL, '') is NULL, which is what GLOB '*' answered. An empty string is a suffix of every value, so it takes instr().
  • $startsWith keeps GLOB. A prefix free of U+0000 lies wholly before the value's first U+0000, or it is not a prefix at all, so the value's cut cannot change that answer (0 of 14 $startsWith probe cases differed at base, including values that start with U+0000). It is also the one text construct an index serves; see H3.
  • The fold is still lower() on both sides, ASCII-only. The negation is still NOT (...) inside the unchanged NULL-safe $notContains wrapper. Nothing is escaped, because instr() and the BLOB suffix have no pattern language, so *, ? and [ are bound as written.
  • Two emitters change: textMatchPredicate / sqliteLengthAwareTextMatch in packages/drivers/driver-sql/src/sql-driver.ts, and RemoteTransport.pushLike in packages/drivers/driver-turso/src/remote-transport.ts. SqliteWasmDriver and TursoDriver's local mode inherit the first. No refusal door, no packages/spec file and no shared conformance table is touched.

Measured at base 9d81af714 (hypotheses H1, H2, H5)

A 292-case probe drove the public find of six faces over 17 stored values (U+0000 at start, middle and end, alone and doubled, multi-byte UTF-8 around it, GLOB and LIKE metacharacters behind it, '' and NULL), on dists built at base:

  • SqlDriver on better-sqlite3 (SQLite 3.53.4);
  • SqliteWasmDriver on sql.js (3.49.1);
  • TursoDriver local;
  • TursoDriver remote over a real @libsql/client :memory: engine (SQLite 3.45.1);
  • InMemoryDriver, and @objectstack/formula matchesFilterCondition as the oracle.

The cases were 5 text operators times 19 comparands (14 without U+0000, 5 with), each bare and under $not; 36 $like and 9 $ilike patterns, each bare and under $not; and 12 $not / $or / $and compositions.

cases (per SQLite face) differ from formula at base at head
text operators, comparand without U+0000, bare 38 of 70 0 of 70
the same under $not 38 of 70 0 of 70
text operators, comparand holding U+0000, bare and under $not 0 of 50 0 of 50
compositions 10 of 12 3 of 12, each with a $like leg
$like / $ilike, bare and under $not 62 of 90 62 of 90 (item 2, unchanged)
  • H1 holds. Per operator at base, bare: $contains 8 of 14, $notContains 8 of 14, $icontains 9 of 14, $endsWith 13 of 14. The four SQLite faces gave byte-identical answer lists on all 292 cases, at base and at head.
  • H2 holds. $startsWith with a comparand without U+0000: 0 of 14 differ, bare and under $not, at base and at head.
  • H5 holds. $icontains: 'É' returns 'CAFÉ' + U+0000 and $icontains: 'é' does not; the fold stays lower() on both sides.
  • InMemoryDriver equals formula on 292 of 292 at base and at head (the control). No face raised on any case, at base or at head.

The constructs themselves were measured with SQL literals, no bind, over 16 comparands without U+0000 (including *?[y, %, _, U+0001 and multi-byte ones), 23 stored values and both folds: instr() > 0, the BLOB suffix, and NOT over each, three-valued against JavaScript includes / endsWith (NULL expected on NULL). That is 2944 cells per engine and 0 disagreements on each of better-sqlite3 3.53.4, sql.js 3.49.1 and libSQL 3.45.1.

H3: the plans

EXPLAIN QUERY PLAN of the SQL SqlDriver compiles, run over create table plan_probe (id integer primary key, v text) with an index on v, on each engine:

filter base (GLOB) head
$contains: 'ab' SCAN plan_probe SCAN plan_probe (instr)
$notContains: 'ab' SCAN SCAN
$icontains: 'Ab' SCAN SCAN
$endsWith: 'ab' SCAN SCAN (BLOB suffix)
$startsWith: 'ab' SEARCH plan_probe USING COVERING INDEX plan_probe_v (v>? AND v<?) the same
$like: 'ab%' SEARCH … USING COVERING INDEX plan_probe_v (v>? AND v<?) the same
$eq: 'ab' (control) SEARCH … (v=?) the same

Identical on better-sqlite3, sql.js and libSQL. H3 holds: the four moved shapes scanned under GLOB already, so no index plan is lost. The remote transport's statements were not put through EXPLAIN; they are the same constructs, pinned by text in turso-20024-glob-stored-nul.test.ts.

Item 2 ($like / $ilike) is not in this PR

Measured, and left for a decision on #20024:

  • There is no NUL-safe pattern primitive to compile to. GLOB and LIKE both stop at U+0000. The way around it I tried first, substituting a sentinel for U+0000 in the value and the pattern, is dead: SQLite's replace() treats a search string whose first byte is 0 as empty and returns its input unchanged. hex(replace('a' || char(0) || 'b', char(0), char(1))) is 610062 on all three engines. Over a grid of 52 patterns, 23 values and both folds (2392 cells per engine), GLOB disagrees with @objectstack/spec matchesLikePattern on 294 cells and the sentinel construct still on 223. instr() does not apply, because the pattern has wildcards, and _ is one character, so a byte-level BLOB program would have to decode UTF-8 lengths.
  • The stored-value cut also hits a pattern without U+0000, which the card's item 2 does not name: at base and at head, 15 of the probe's 23 such $like patterns differ from formula on every SQLite face. For example, $like: 'a' returns 'a' + U+0000 + 'b', and $like: '' returns U+0000 + 'z'. No refusal can reach this half, because it depends on stored data.
  • A refusal is a cross-face contract question. The dangling-escape refusal of a $like pattern is one spec predicate (hasDanglingLikeEscape) applied at every face's refusal door. The same shape for U+0000 needs packages/spec and the refusal doors that engine + driver-sql: seven filter-refusal doors ignore the #8220 provenance mark, so a refusal inside a policy read-scope subtree relays the policy field (and some comparands) in its 400: the #7929 redaction is not applied #20020 is working in, and what Postgres and MySQL do with such a pattern is not measured here (no live server). A SQLite-only refusal would make SQLite the one face that refuses what driver-memory answers.

The options, with the measured costs, are in the os-dev-report on #20024.

Compile surfaces (references/compile-surfaces.md, re-verified with its grep at this head)

# face status evidence
1 driver-sql textMatchPredicate SQLite arm (sql-driver.ts) touched sql-driver-20024-glob-stored-nul.test.ts (better-sqlite3). driver-sqlite-wasm inherits it and is pinned by sqlite-wasm-20024-glob-stored-nul.test.ts; driver-turso local mode by turso-20024-glob-stored-nul.test.ts.
1 the same function's Postgres and MySQL arms not touched, NOT MEASURED No live server in this container.
1 driver-sql likePatternPredicate SQLite arm not touched, measured still cut Item 2, above.
2 turso RemoteTransport.pushLike (remote-transport.ts) touched Pinned on makeLibsqlSqliteStub AND on a real @libsql/client file::memory: engine, beside local mode, all three to the JavaScript rows. A real Turso server and its wire encoding of U+0000 are NOT MEASURED.
2 turso RemoteTransport.pushLikePattern not touched, measured still cut On the real libSQL engine, in the probe.
3 service-analytics compileScopedFilterToSql not touched, NOT MEASURED Outside this claim; its SQLite text arm is #20025's.
4 service-analytics lowerAnalyticsWhere not touched, NOT MEASURED Outside this claim, as face 3.
5 formula matchesFilterCondition measured unaffected It is the oracle in all three new suites and in the probe.
half objectql applyHaving measured unaffected Over 9 stored values, 6 of them holding U+0000, 30 cases (5 operators times 6 comparands without U+0000): 29 equal formula; $icontains: '' is refused, the pre-existing refusal driver-sql shares.
thawed driver-memory checkCondition measured unaffected 292 of 292 probe cases equal formula, at base and at head.
thawed driver-mongodb translateFieldOperators not touched, NOT MEASURED No mongod in this container.

Tests

  • New suites, each case pinned literally: sql-driver-20024-glob-stored-nul.test.ts (74 tests), sqlite-wasm-20024-glob-stored-nul.test.ts (24), turso-20024-glob-stored-nul.test.ts (26, three transports per case). The driver-sql and sqlite-wasm suites also check every case against formula; the turso cases are a subset of the driver-sql rows. They cover the four moved operators, $startsWith (H2), the empty comparand, and $not / $or / $and over each moved shape with the '' and NULL rows in every answer set. The compile block pins the new SQL and raw bindings, the empty-$endsWith route, and $startsWith's GLOB plus its escaped pattern byte for byte.
  • pnpm --filter @objectstack/driver-sql exec vitest run --maxWorkers=2 at 667eddc47: 183 files passed, 11 skipped (live-dialect cells); 2811 tests passed, 170 skipped.
  • driver-turso 65 of 65 files, 1471 of 1471 tests, and driver-sqlite-wasm 32 of 32 files, 606 of 606 tests, at cf0590562, whose code is byte-identical to 667eddc47 (the one later commit adds only the changeset).
  • typecheck (tsc --noEmit) exits 0 in all three packages, and tsc --listFiles shows each program includes its new suite.
  • ESLint, narrowed: --no-inline-config --format json over the 15 changed TS files reports 15 files, 0 errors, 0 warnings, none ignored (every file resolves a config through --print-config). eslint.config.mjs never enables type-aware linting (no parserOptions.project, as its own comment says), so this diff cannot move the verdict of a file it does not touch. The full pnpm lint is CI's.

Respelled assertions (markers the moved plans broke)

file old new
driver-sql/src/sql-driver-19999-glob-nul-comparand.test.ts $contains 'a*b' compiles to `v` GLOB ? with *a[*]b*; $endsWith binds *ab; $icontains compiles lower(v) GLOB lower(?); $notContains matches NOT GLOB only $startsWith keeps `v` GLOB ? with ab*; the other four compile without GLOB and bind a*b raw
driver-sql/src/sql-driver-15683-temporal-text-operator-gate.test.ts text column toMatch(/LIKE|GLOB/); negatives without instr( toMatch(/LIKE|GLOB|instr\(/); instr\( added to both negatives
driver-sql/src/sql-driver-17343-multi-valued-boolean-membership.test.ts REAL_PREDICATE.sqlite /LIKE|GLOB|json_each\(/; control toMatch(/LIKE|GLOB/); negative without instr( /LIKE|GLOB|instr\(|substr\(CAST\(|json_each\(/; control /LIKE|GLOB|instr\(/; instr\( added to the negative
driver-sql/src/sql-driver-17590-json-column-membership.test.ts scalar column toMatch(/LIKE|GLOB/) and toMatch(/GLOB/); rest-of-family toMatch(/LIKE|GLOB/); "not a pattern match" not.toMatch(/LIKE|GLOB/) /LIKE|GLOB|instr\(/, /instr\(/, /LIKE|GLOB|instr\(|substr\(CAST\(/, and the negative gains instr\(|substr\(CAST\(
driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts lower(name) GLOB lower( and *acme*; name GLOB instr(lower(name), lower('acme')) > 0; instr(name, 'acme') > 0; neither contains GLOB
driver-sql/src/sql-driver-text-case-conformance.test.ts "sqlite: GLOB, case-exact": $contains toMatch(/GLOB/), $icontains lower(…) GLOB lower(; numeric-column control toMatch(/LIKE|GLOB/) $contains instr(, $icontains instr(lower(…), lower(, $startsWith GLOB; control /LIKE|GLOB|instr\(/; instr\( added to both negatives
driver-turso/src/turso-19999-glob-nul-comparand.test.ts remote $contains 'a*b' sends "v" GLOB ? remote $startsWith 'a*b' sends "v" GLOB ?
driver-turso/src/remote-transport-text-predicates.test.ts lower("name") GLOB lower(?); "name" GLOB ?; $contains * / ? / [ / a*b bind *[*]*, *[?]*, *[[]*, *a[*]b* instr(lower("name"), lower(?)) > 0; instr("name", ?) > 0; the escape class is pinned on $startsWith ([*]*, [?]*, [[]*, a[*]b*), plus a new case that $contains 'a*b' binds a*b raw
driver-turso/src/remote-transport-null-comparand-refusal.test.ts $contains: true binds *true* binds true
driver-turso/src/remote-transport-undefined-comparand-refusal.test.ts $contains: 'w' binds *w* binds w

Before-red and ablations (one-off; no test left behind)

Both ran from committed state through scripts/ablation-replace.mjs (anchor counts and blob hashes checked on disk, restore proven as blob == HEAD with git diff HEAD empty), with driver-sql rebuilt and scripts/ablation-dist-preflight.mjs confirming the mutation in dist/ before the suites ran, because driver-sqlite-wasm and turso's local mode read that dist.

  • Ablation 1, the dispatch reverted in both emitters (if (shape !== 'starts' || text.includes(NUL_CHARACTER)) back to base's if (text.includes(NUL_CHARACTER))), which is base behaviour. Predicted by simulating GLOB's cut over each suite's cases, then observed exactly: driver-sql 24 failed and 50 passed of 74 (22 row cases plus the moved-shapes and empty-$endsWith compile pins; the $startsWith pin stayed green); sqlite-wasm 16 failed and 8 passed of 24; turso 18 failed and 8 passed of 26. In the 12 turso row diffs whose three columns printed in full, local, remote and remoteLibsql were all wrong the same way; the base probe had already found local mode and remote-over-libSQL byte-identical to the other SQLite faces on all 292 cases.
  • Ablation 2, the empty-comparand route removed (shape === 'ends' && text !== '' to shape === 'ends', both emitters). Predicted and observed: driver-sql 3 failed of 74 ($endsWith '', $not $endsWith '', the empty-$endsWith compile pin), sqlite-wasm 2 of 24, turso 3 of 26.
  • After each: git status --porcelain empty, driver-sql rebuilt, and preflight reads the guard back in dist/ with the tree clean.

Gates (at 667eddc47)

  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 62 commands from the real diff, the same 62 the PM list named. All 62 were run.
    • check:dual-build-cjs-loads, check:lean-entry-closure and check:type-check-debt first answered PREREQUISITE NOT MET (exit 3). They answered 0 after turbo run build --filter='./packages/*' --filter='./packages/*/*' (72 tasks). check:dts-closure, check:published-files and check:sourcemap-no-sources-content were re-run over that build and answered 0.
    • --ran: "62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN".
  • check-changeset-no-major: "This diff introduces no major bump." check-adr-0087-registration: "this PR adds no declared-breaking changeset (1 non-breaking changeset(s) seen)."
  • node scripts/check-issue-citations.mjs --base 9d81af714: 12 citations across 2 files, all resolve.
  • pnpm check:driver-conformance reads the same before (on a git archive of 9d81af714) and after: 50 covered cell(s), 0 in the DEBT ledger, 0 exempt.
  • The roster gates whose roster sits beside these paths also exit 0: check-changeset-fixed, check:authz-resolver, check:error-code-casing, check:filter-alias-parity.
  • pnpm check:nul-bytes passes, and a control-byte scan of the changed files finds none. U+0000 is spelled String.fromCharCode(0x00) throughout.

Acceptance notes (not filed; for the seat)

  1. Item 2 of driver-sql / driver-turso (SQLite faces): GLOB still cuts a STORED value at its first U+0000, so a NUL-free $contains / $notContains / $icontains / $endsWith answers wrongly on it; a $like / $ilike pattern holding U+0000 is cut the same way #20024 is open and needs a decision (above; options in the os-dev-report).
  2. Observation, not filed: packages/plugins/plugin-auth/src/auth-contains-filter.test.ts says in its docblock that the SQL family "emits GLOB on the SQLite dialects now". For $contains that is instr() after this PR. It is prose in a test outside this claim; no assertion reads it.

Generated by Claude Code

…eads the whole stored value

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…ved contains/ends plans break

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…every SQLite face; respell the remote GLOB pins

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…for the SQLite stored-value cut

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/xl documentation Improvements or additions to documentation tests tooling labels Sep 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

4 anchor(s) derived from 2 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • the SDK route bridge reached 60 of 215 client-bound route-ledger rows — the other 155 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 155: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 100 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 13 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 7766b62282f456ef04fd37304658b7ecd5a0fff9 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from e174d034001090f261b4d7e2759013cfcee98abf — the merge of head 667eddc476477d9b15bf05f4b18f02b684315410 into base 7766b62282f456ef04fd37304658b7ecd5a0fff9, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin e174d034001090f261b4d7e2759013cfcee98abf && git checkout e174d034001090f261b4d7e2759013cfcee98abf
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 7766b62282f456ef04fd37304658b7ecd5a0fff9 667eddc476477d9b15bf05f4b18f02b684315410 && git checkout -B drift-repro 7766b62282f456ef04fd37304658b7ecd5a0fff9 && git merge --no-ff 667eddc476477d9b15bf05f4b18f02b684315410

node scripts/docs-audit/affected-docs.mjs --json 7766b62282f456ef04fd37304658b7ecd5a0fff9

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 667eddc476477d9b15bf05f4b18f02b684315410

Scope read: card #20024 (body and all comments), the PR body, and the diff 9d81af714..667eddc47 (merge base verified; 16 files, +915/−122, 4 commits; the remote tip equals this head).

Measured in the reviewer's own detached worktree at this head, which was removed afterwards. The engines are the lockfile's, read with sqlite_version():

  • better-sqlite3 13.0.3 = SQLite 3.53.4;
  • sql.js 1.14.1 = 3.49.1;
  • @libsql/client 0.17.4 = 3.45.1, a real file::memory: engine behind TursoDriver in remote mode.

① Derived judgments

  • Accept set unchanged: RIGHT. The diff adds no throw and removes none. Of 445 cases per face through the public find, 3 raise at head and 3 with the base emitters, all the pre-existing $icontains: '' refusal. InMemoryDriver raises the same 3.
  • Public surface unchanged: RIGHT. sqliteLengthAwareTextMatch and NUL_CHARACTER are absent from driver-sql's dist/index.d.ts. In turso's .d.ts, the name appears only inside the docblock of private pushLike.
  • Answer set of the four moved operators, through the public find: RIGHT.
    • The grid: 5 operators × 29 comparands (5 of them holding U+0000) × {bare, $not, $not $not}, plus 10 $or / $and / nested-$not compositions, over 21 stored values. Those values put U+0000 at the start, middle and end, and add multi-byte UTF-8, GLOB metacharacters, '' and NULL.
    • The faces: SqlDriver/better-sqlite3, SqliteWasmDriver/sql.js, and TursoDriver local and remote over real libSQL.
    • At head, 0 of 445 differ from @objectstack/formula matchesFilterCondition on each face, and the four faces' answer lists are byte-identical. The InMemoryDriver control also differs on 0 of 445.
    • With the dispatch reverted in both emitters, three faces differ on 141 of 445, alike on every one: $contains, $notContains and $icontains 33/87 each, $endsWith 36/87, $startsWith 0/87, compositions 6/10. The remote face at base is evidenced by the ablated turso suite, where all three columns are wrong the same way.
    • The zero-length / NULL edge that failed the driver-sql (SQLite faces): a $contains / $startsWith / $endsWith comparand holding U+0000 is cut at the NUL by glob(), so the filter answers wrongly; one that starts with U+0000 makes $contains / $endsWith match every row #19999 round-0 review is hunted and closed. $not $endsWith '' returns only the NULL row. Every other $not and $not $not form keeps the '' row exactly as formula does, on all four faces.
  • The constructs, raw three-valued, as SQL literals: 9108 cells per engine, 0 disagreements with JavaScript includes / endsWith / startsWith under the ASCII fold, with NULL answering NULL, on all three engines. The bound form gives 11232 cells and 0 disagreements on better-sqlite3 and libSQL. Base GLOB over the NUL-free comparands differs on 79 of 3726, identically on all three engines.
  • $startsWith without U+0000 stays GLOB, byte for byte: RIGHT. Its arm is unchanged in the diff. Measured: 0 of 87 differ at base or head, including stored values that begin with U+0000.
  • Case folding: RIGHT. The fold is ASCII-only and runs past U+0000 (hex(lower('A'||char(0)||'B')) = 610062 on all three engines). É, é, ß and S agree with formula.
  • Plans: RIGHT. EXPLAIN QUERY PLAN over an indexed text column is identical on the three engines, at base and at head:
    • $contains, $notContains, $icontains and $endsWith are a SCAN under both;
    • $startsWith 'ab' and $like 'ab%' are a covering-index SEARCH under both.
    • No index SEARCH is lost.
  • Respelled pins: none weakened. Every positive LIKE|GLOB marker in the 15683, 17343, 17590 and text-case-conformance suites gains instr\(, and so does every negative one. The 17590 one_str pin tightens from /GLOB/ to /instr\(/. The pins in the retired-operator and remote-transport suites become exact instr(…) > 0 pins, and the escape class moves to $startsWith. No pin outside the diff reads driver-sql's emitter through a negative that matches only LIKE|GLOB.
  • Red, then green, in the reviewer's worktree:
    • At head: driver-sql 273 passed / 8 skipped (the 20024 suite 74/74), sqlite-wasm 24/24, turso 168 passed (the 20024 suite 26/26).
    • Ablation 1, the dispatch reverted: 24/74, 16/24 and 18/26 failed.
    • Ablation 2, the empty-ends route removed: exactly the $endsWith '' rows failed (3, 2 and 3).
    • Both match the PR's numbers exactly. The restore was proven blob-equal to HEAD.
  • CI at this head: 34 runs, 0 failures, and every required context is success. The only skips are Build Docs, Console Pin Gate and Packed-tarball smoke (opt-in).

② Semver level

patch for @objectstack/driver-sql, @objectstack/driver-sqlite-wasm and @objectstack/driver-turso, with Clause-②: no: RIGHT. Wrong answer sets move toward formula; nothing is refused or newly accepted, and nothing new is exported. The three packages are one fixed group.

③ Boundary flags

Changeset: every sentence at head is TRUE, including:

  • the mechanism sentence;
  • "nothing raised";
  • "all four answered alike";
  • the four symptom bullets;
  • "for every comparand" and the empty-$endsWith route;
  • "returns the rows driver-memory and @objectstack/formula return … under $not, $or and $and";
  • the literal * / ? / [ sentence, the ASCII-only fold, and $notContains keeping the NULL row.

In "What does not change", these hold:

  • $startsWith is byte for byte unchanged, and no index is lost (both measured).
  • The Postgres and MySQL arms are untouched, per the diff. Their behaviour is not claimed.
  • The $like / $ilike limitation is stated truthfully: through find they still differ from formula, identically on every face.

Docblocks in sql-driver.ts and remote-transport.ts: every factual sentence is TRUE (measured on three engines).

PR body: TRUE where measured. The author's exact figures (292 cases, "38 of 70", "15 of 23", 2944 cells) are UNMEASURED as figures, but every direction is confirmed and the ablation counts match. Faces 3 and 4, the PG/MySQL arms, a real Turso server and mongod are correctly declared NOT MEASURED.

Out-of-scope observation (not a FAIL item): the plugin-auth auth-contains-filter.test.ts docblock sentence "the SQL family emits GLOB on the SQLite dialects now" is now false for $contains. The same stale prose sits in three untouched test docblocks in the touched packages: sqlite-wasm-dialect-identity.test.ts line 14, sqlite-wasm-icontains-and-retired-operators.test.ts (its docblock and one case title), and turso-local-remote-text-parity.test.ts lines 21–24 and 190. It is all prose; every assertion there holds and is green in Test Core.

Open question (item 2, $like / $ilike over U+0000): not for this record to rule. The changeset states the limitation truthfully, and the dangling-escape precedent that option A cites (hasDanglingLikeEscape) is real.

Implemented-by: claude/issue-20024-sqlite-glob-stored-nul
Reviewed-by: session_01Bvd69VPa6puiNzzPUroDBx

VERDICT: PASS

Isolated reviewer: a contract-review-tier subagent, fed only the card, the PR and AGENTS.md; adopted by the seat.

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review September 24, 2026 22:52
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Sep 24, 2026
Merged via the queue into main with commit adbbc5d Sep 24, 2026
35 of 36 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-20024-sqlite-glob-stored-nul branch September 24, 2026 23:08
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Sep 28, 2026
…le stored value past a U+0000 (objectstack-ai#20024) (objectstack-ai#20140)

Fixes objectstack-ai#20024

Clause-②: no

Item 2 (ii) of objectstack-ai#20024, direction C (landing record 5823785768). On the
SQLite faces, a `$like` / `$ilike` pattern without U+0000 over a stored
value holding U+0000 now gets the answer `likePatternToRegexSource`
gives, which is also what `@objectstack/formula` and `driver-memory`
return. Item 1 landed as PR objectstack-ai#20038 and item 2 (i) as PR objectstack-ai#20124 (objectstack-ai#20041).
This is the card's last item, so the PR closes it.

`patch` for `@objectstack/driver-sql`, `@objectstack/driver-sqlite-wasm`
(it inherits the fix) and `@objectstack/driver-turso`. Nothing that was
accepted is refused now: the refusal doors, `hasNulInLikePattern` and
`hasDanglingLikeEscape` are untouched. The grid, plan and cost numbers
below were taken at `b424b90f7c` or earlier. Head `89e02a2521` differs
from `b424b90f7c` by one changeset line only, and its emitter code is
byte-identical to `7434e6646e`; every commit since then changed
changeset prose only.

## The defect, measured at base `fe677aeeed`

`glob()` reads the stored value as a C string, only up to its first
U+0000. The grid has 59 stored values (27 holding U+0000, leading,
interior, trailing, doubled, next to multi-byte and U+0001 characters,
plus NULL). It runs 130 queries against them: 81 `$like` and 27 `$ilike`
bare patterns, and 22 `$not` / `$or` / `$and` compositions. That is 7670
cells per face, each compared with `formula`. The four faces answered
identically:

| stored value | cells per face | base: differ from `formula` | head:
differ from `formula` |
|---|---|---|---|
| holds U+0000 (26 values in the Basic Multilingual Plane) | 3380 | 359
| **0** |
| no U+0000, in the Basic Multilingual Plane (28 values) | 3640 | 0 | 0
|
| NULL | 130 | 0 | 0 |
| outside the Basic Multilingual Plane, no U+0000 (3 values) | 390 | 20
| 20 (older than this PR, see H3) |
| outside the Basic Multilingual Plane, with U+0000 (1 value) | 130 | 6
(6 against a code-point oracle) | 2 (0 against a code-point oracle) |

The four faces are `SqlDriver` on better-sqlite3 (SQLite 3.53.4),
`SqliteWasmDriver` on sql.js (3.49.1), `TursoDriver` local, and
`TursoDriver` remote over a real `@libsql/client` `file::memory:` engine
(3.45.1). At base, 74 of the 108 bare patterns got at least one row
holding U+0000 wrong. Some examples:

- `$like: 'a'` returned `'a'` + U+0000 + `'b'`;
- `$like: ''` returned U+0000 + `'z'`;
- `$like: '_'` missed a lone U+0000;
- `$ilike: 'A_B'` missed `'a'` + U+0000 + `'b'`.

**Base vs head, every stored value without U+0000 and NULL: 0 of 16640
cells moved, on any face.**

## What changes

`likePatternPredicate`'s SQLite arm in `sql-driver.ts` now calls
`sqliteLikePatternMatch`. `RemoteTransport.pushLikePattern` in
`remote-transport.ts` restates the same rule for the remote transport.
The PR's own parity test holds the two to the same predicate text and
bindings, the precedent `pushLike` already follows. Sharing one compiler
would need a new export from `driver-sql`'s `index.ts`, which is outside
this claim's file surface. How a pattern compiles:

- **A literal prefix followed only by `%`** (`'ab%'`, `'%'`, `'%%'`)
keeps the bare `col GLOB ?` with the same binding as before. A prefix
free of U+0000 cannot be changed by the cut, the same argument
`$startsWith` stands on.
- **Every other pattern** compiles to `CASE WHEN instr(CAST(col AS
BLOB), X'00') > 0 THEN (a recursive CTE that replaces each U+0000 with
one stand-in character, then GLOB) ELSE col GLOB ? END`.
- A value without U+0000 takes the `ELSE` arm: the same `GLOB` as
before.
- NULL also takes the `ELSE` arm and stays NULL, so the NULL-safe `$not`
wrapper composes unchanged.
- **A case-exact pattern with a literal prefix** leads with `col GLOB
'PREFIX*'`, where PREFIX is the text before the first unescaped
wildcard. Every value the pattern matches satisfies that conjunct, cut
or not, and it keeps the index range. `$ilike` has no index range to
keep, since `lower(col)` is not an indexed column, so it gets no
conjunct.

**Why the rewrite is exact.** The stand-in is the smallest code point
from U+0001 up that is not a literal character of the pattern, not an
ASCII letter and not a surrogate. The pattern holds no U+0000, because
`hasNulInLikePattern` refuses it at every door. So a U+0000 in the value
can only be matched by `%` or `_`. The stand-in is one character, as
U+0000 is, and no literal position of the pattern matches it, just as
none matched U+0000. Because it is not a letter, the `$ilike` fold
cannot turn it into one. So the rewritten value matches exactly when the
original does.

**H2 is measured, not assumed.** On all three engines, `replace(v,
char(0), char(1))` returns its input unchanged, `GLOB` and `LIKE` cut at
U+0000, `lower()` keeps the bytes past U+0000, and `instr()` over BLOB
finds U+0000. A correlated recursive CTE inside a scalar subquery works
on all three and splices every U+0000, with NULL staying NULL. The
earlier sentinel measurement (223 of 2392 cells wrong) is consistent
with this: `replace()` never reached a U+0000 there.

## H3: characters outside the Basic Multilingual Plane under `_` (older
than this PR, not widened)

The spec's regex has no `u` flag, so `_` in `formula` and
`driver-memory` matches one UTF-16 unit, while `GLOB`'s `?` matches one
code point. At base and at head alike, 20 of 390 cells over such values
without U+0000 differ from `formula` on every SQLite face (17 queries).
For example, `$like: '_'` returns a stored '😀' on SQLite and not in
`formula`. This PR's U+0000 arm uses `GLOB` too, so it follows the
code-point reading, consistent with the arm for values without U+0000.
On the astral value holding U+0000, 0 of 130 head cells differ from a
code-point oracle (the spec's regex compiled with the `u` flag). The
divergence is reported to the seat as an out-of-scope finding and not
fixed here.

## H4: plans and cost

`EXPLAIN QUERY PLAN` of the statements the real compilers emit (the
`SqlDriver` compile, and the remote statement captured from a recording
libSQL client) over `plan_probe(id integer primary key, v text)` with an
index on `v`. Identical on better-sqlite3, sql.js and libSQL:

| filter | before | after |
|---|---|---|
| `$like 'ab%'` | SEARCH USING COVERING INDEX (v greater-than ? AND v
less-than ?) | same, same statement |
| `$like 'ab_'`, `'ab%cd'`, `'abc'`, `'a%b%'` | SEARCH USING COVERING
INDEX (range) | the same SEARCH, plus a correlated scalar subquery |
| `$like '%ab'`, `'_b'`, `''` | SCAN | SCAN, plus a correlated scalar
subquery |
| `$like '%'`, `$ilike 'ab%'` | SCAN | SCAN, same statement |
| `$ilike 'ab_'` | SCAN | SCAN, plus a correlated scalar subquery |
| `$eq`, `$startsWith` (controls) | SEARCH | unchanged |

Cost is the median of 7 runs of `count(*)` over 10,000 rows, base `GLOB`
against head. This is a shared box, so read the ratios:

| rows holding U+0000 | `$like '%ab%c'` | `$like '_b%'` | `$like 'ab_%'`
| `$like 'ab%'` | `$ilike '%AB%C'` |
|---|---|---|---|---|---|
| 0% | 1.4–1.8x | 1.7–2.4x | 1.1–2.2x | 1.0x | 1.3–1.4x |
| 10% | 4.2–4.7x | 5.4–7.8x | 3.3–3.7x | 0.9–1.0x | 2.4–2.8x |
| 100% (1–3 each) | 25–37x (23–34 ms) | 26–44x (25–31 ms) | 23–33x (5–7
ms) | 0.7–1.1x | 12–14x (25–35 ms) |

The cost for values without U+0000 is the per-row `instr()` guard. A
text-`instr` spelling of it is also correct on all three engines, but it
measured slower than the BLOB one. Every head count equals `formula`'s
count, and base counts differ wherever U+0000 rows exist.

## H5: the remote face, executed

`TursoDriver` remote over a real `@libsql/client` engine runs the whole
grid in the probe (0 of 3380 cells differ at head) and the whole pin
suite in CI. Its predicate text and bindings equal the local compiler's,
modulo identifier quoting. A real Turso server, and its wire encoding of
a string holding U+0000, are NOT MEASURED.

## H6: two pending changesets, two DELIBERATE CORRECTIONS (please
confirm)

This PR makes one sentence false in each of two pending changesets, and
it corrects each sentence in place. One line changes in each file, and
every other line of both files is byte-identical.

- `Check Changeset` goes RED on both by design.
`scripts/check-empty-changeset.mjs`'s foreign-changeset rule is
content-blind: any `M` row on a changeset present at the merge base is
refused, whichever card owns it.
- Both are the **DELIBERATE CORRECTION** class. Its remedy is to confirm
the correction here. ⛔ Restoring either file would put a false sentence
back.

1. `.changeset/20024-sqlite-glob-stored-nul.md` is this card's own item
1 entry. The last bullet changed (line 29 of 29):
- Before: "- `$like` and `$ilike` still compile to `GLOB` on SQLite, so
they still read a stored value only up to its first U+0000. A pattern
holding U+0000 is no longer cut there: every driver that answers `$like`
now refuses it (`INVALID_FILTER` / 400), by its own entry in this
release."
- After: "- `$like` and `$ilike` are outside this entry. Two other
entries in this release cover them: on SQLite they now read the whole
stored value as well, and every driver that answers `$like` refuses a
pattern holding U+0000 (`INVALID_FILTER` / 400)."
2. `.changeset/20041-like-nul-pattern-refused.md` is objectstack-ai#20041's entry. The
first bullet under **Not changed here** changed (line 36 of 44). The
seat's claim amendment 5830697185 on objectstack-ai#20024 (ruling A) added this bullet
to this PR's surface.
- Before: "- A pattern without U+0000 matched against a STORED value
that holds U+0000 still differs from `formula` on the SQLite faces,
because `GLOB` also reads the stored value only up to its first U+0000.
No refusal of the pattern can reach that half."
- After: "- A pattern without U+0000 matched against a STORED value that
holds U+0000 is not refused: it is well formed, and on the SQLite faces
it reads the whole stored value, by its own entry in this release."
   - The new sentence is true at head `89e02a2521`:
- The refusal doors read only the pattern (a non-string pattern, a
dangling escape, or a pattern holding U+0000).
- The three pin suites pass there: 65 of 65, 22 of 22 and 23 of 23. They
include patterns without U+0000 over stored values holding U+0000 on
every SQLite face, and every such case returns `formula`'s rows and none
is refused.
     - "Its own entry" is `.changeset/20024-like-stored-nul.md`.

The new entry is `.changeset/20024-like-stored-nul.md`.

## Tests (at `7434e6646e`; the code is identical at head)

- New suites:
- `sql-driver-20024-like-stored-nul.test.ts`: 65 of 65 pass. It has 29
literal cases, each pinned and checked against `formula`, a 67-pattern
grid held to `formula` cell by cell, the exact compiled statements, and
an `EXPLAIN QUERY PLAN` pin.
- `sqlite-wasm-20024-like-stored-nul.test.ts`: 22 of 22 pass (19 literal
cases, the grid, a plan pin, and the stored-bytes premise).
- `turso-20024-like-stored-nul.test.ts`: 23 of 23 pass. It runs local,
remote over the stub, and remote over a real libSQL engine, plus the
grid and the emitter-parity case. `driver-turso` does not depend on
`formula`, so its grid is held to the spec's `matchesLikePattern`.
- Full suites:
- `pnpm --filter @objectstack/driver-sql exec vitest run
--maxWorkers=2`: 187 files passed and 11 skipped; 3089 tests passed and
170 skipped.
  - `@objectstack/driver-sqlite-wasm`: 34 of 34 files, 644 of 644 tests.
  - `@objectstack/driver-turso`: 70 of 70 files, 1686 of 1686 tests.
- No existing pin moved: before the new suites were added, all three
full suites were green against the new emitters.
- `pnpm --filter … run typecheck` exits 0 for all three packages, and
`tsc --listFiles` counts each new suite once.
- Ablations, each with a prediction made from the base statement or a
simulation, a mutation proved on disk through
`scripts/ablation-replace.mjs`, a `driver-sql` rebuild, and
`ablation-dist-preflight` (marker present, then `--absent` after
restore). Each restore was proved as the blob hash equal to HEAD, with
`git status --porcelain` empty.
- (1) Dispatch reverted in both emitters, which gives the base
statement. Predicted and observed: driver-sql 25 failed / 40 passed,
sqlite-wasm 15 / 7, turso 16 / 7.
- (2) Stand-in fixed at U+0001 in both emitters. Predicted and observed:
4 / 61, 2 / 20, 2 / 21. The failures are the U+0001 cases, the grid, and
the stand-in statement pin.
- (3) Remote emitter only. Predicted and observed: turso 17 failed / 6
passed, including the parity test.

## Gates (the full union at `b424b90f7c`; the changeset gates re-run at
`89e02a2521`)

At `89e02a2521`, the head that adds the second correction:

- `node scripts/check-empty-changeset.mjs --base origin/main` exits 1.
This is expected and names both files: "This PR changes a changeset it
did not add: .changeset/20024-sqlite-glob-stored-nul.md present on the
merge base and CHANGED by this PR ...
.changeset/20041-like-nul-pattern-refused.md present on the merge base
and CHANGED by this PR". Both are the H6 DELIBERATE CORRECTIONS.
- `node scripts/check-changeset-no-major.mjs --base origin/main
--event`, over this body, exits 0: "This diff introduces no `major`
bump.", and "LEVEL AXIS: this PR declares clause-② `no`, so no package
here is declared to have grown a published surface" (declaration line
`Clause-②: no`, no direction arm).
- `node scripts/check-adr-0087-registration.mjs --base origin/main`
exits 0: "check-adr-0087-registration: this PR adds no declared-breaking
changeset (3 non-breaking changeset(s) seen)."
- `node scripts/check-issue-citations.mjs --base fe677ae` exits 0: 7
citations across 2 files resolve.

At `b424b90f7c`:

- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derives 62 commands. All 62 were run and
reconciled with `--ran`: 62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN.
- Every command exits 0 except `node scripts/check-empty-changeset.mjs
--base origin/main`, which exits 1 for the H6 correction above.
- `check:dual-build-cjs-loads`, `check:lean-entry-closure` and
`check:type-check-debt` first exited 3 (PREREQUISITE NOT MET). They exit
0 after `turbo run build` over `./packages/*` and `./packages/*/*` (72
tasks). The dist-reading gates were re-run over that build and are green
too.
- `check:driver-conformance`: 50 covered cells, 0 DEBT, 0 exempt.
- `node scripts/check-issue-citations.mjs --base fe677ae`: 7
citations resolve.
- The rosters beside these paths are green: `check-changeset-fixed`,
`check:authz-resolver`, `check:error-code-casing`,
`check:filter-alias-parity`.
- ESLint, narrowed: `eslint --no-inline-config --format json` over the 5
changed `.ts` files gives 5 files, 0 errors, 0 warnings, 0 ignored.
- The population is every changed `.ts` file, each resolving a config
under `--print-config`.
- `eslint.config.mjs` never enables type-aware linting, so no untouched
file's verdict can move. The full `pnpm lint` run is CI's.

## Compile surfaces

1. `driver-sql` `likePatternPredicate`, SQLite arm: TOUCHED. Measured on
better-sqlite3 3.53.4. `SqliteWasmDriver` (sql.js 3.49.1) and
`TursoDriver` local inherit it, and each is measured by the grid and by
its own suite. The Postgres and MySQL arms are not touched and NOT
MEASURED (no live server here).
2. turso `RemoteTransport.pushLikePattern`: TOUCHED. Measured on
`makeLibsqlSqliteStub` and on a real `@libsql/client` engine (3.45.1). A
real Turso server is NOT MEASURED.
3. and 4. service-analytics `compileScopedFilterToSql` /
`lowerAnalyticsWhere`: not touched. `$like` / `$ilike` are in their
unevaluated set (`preview-unevaluable-operator.test.ts`), so these faces
do not compile them. Rows are NOT MEASURED.
5. `formula` `matchesFilterCondition`: not touched; it is the oracle.
- Half face: objectql `having-filter`: not touched, and `$like` does not
occur in `having-filter.ts`.
- Thawed faces:
- `driver-memory` `checkCondition`: not touched. Measured on the same
grid: 0 of 7670 cells differ from `formula`.
  - `driver-mongodb`: not touched, NOT MEASURED (no mongod).

## Acceptance notes

- H3's divergence outside the Basic Multilingual Plane is reported to
the seat with its measured reach, and is not fixed here.
- The objectstack-ai#20041 changeset: its line 32 ("every `$like` / `$ilike` pattern
without U+0000 answers exactly as before") is scoped to objectstack-ai#20041's own
change and is left as it is. The corrected line 36, in the same section,
states what this PR changes for stored values.
- Draft PR objectstack-ai#20104 edits `RemoteTransport` elsewhere in
`remote-transport.ts`. This PR's hunk is inside `pushLikePattern` only.

Written by `session_01Bvd69VPa6puiNzzPUroDBx` (os-dev, `mode:subagent`)
for the `domain:engine` seat.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/xl tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant