Skip to content

fix(driver-turso): the remote canonical backfill never writes a julian misreading over a stored value - #19719

Merged
huangyiirene merged 1 commit into
mainfrom
claude/issue-6009-turso-julian-guard
Sep 22, 2026
Merged

huangyiirene merged 1 commit into
mainfrom
claude/issue-6009-turso-julian-guard

Conversation

@huangyiirene

Copy link
Copy Markdown
Collaborator

Fixes #6009

Clause-②: no

The sequel to PR #19688, which fixed the driver-sql (Knex) half of this card at squash 40626bdca6. The REMOTE path is a separate module that builds its own statements and did not inherit the guard.

The defect

RemoteTransport.mapFieldTypeToSQL declares every temporal column TEXT, so a Field.datetime / Field.time column can hold a digits-only string such as '2026', '86400', '12' — or the keyword 'now'. SQLite's time-value grammar accepts a bare number as a JULIAN DAY and now as the wall clock, so for those two shapes strftime answers confidently rather than returning NULL, and the coalesce(strftime(...), col) that is supposed to preserve unparseable values never fires. The convergence UPDATE in backfillRemoteCanonicalColumn wrote that answer over the stored bytes, and no later run can get them back.

Measured here on better-sqlite3 13.0.3 / SQLite 3.53.4, column declared TEXT:

'2026'  -> -4707-06-11T12:00:00.000Z    'now' -> the wall clock, now
'86400' -> -4476-06-15T12:00:00.000Z    '12'  -> -4713-12-06T12:00:00.000Z

In a Field.time column the same '2026' becomes '12:00:00' — a wall clock nobody wrote. Both are pinned.

What changed

All of it inside packages/drivers/driver-turso/src/. No packages/drivers/driver-sql/ file is touched, and neither sqliteCanonicalDatetimeSql nor sqliteCanonicalTimeSql moves — the 2026-08-03 cloud#1005 ruling that refused a heuristic in the shared read expression is untouched, and #19688's pins that those expressions still misread a bare number stay green.

  • TursoDriver hands its own sqliteNonTemporalTextSql across the module boundary, beside the canonical expression it already hands over, as { canonical, nonTemporalText }. The predicate is protected on SqlDriver and stays that way; nothing is copied, so remote and local cannot drift about which bytes are safe to overwrite.
  • The convergence UPDATE gains and not (guard) — the local twin's exclusion, same construction.
  • Withheld rows block the canonical mark. This is the half that keeps every query answer identical, and here it costs no second gate: a withheld row is by construction col IS NOT canonical, so it is already inside the probe's residual, and the mark already requires residual === 0. Withholding alone would not do it — a marked column drops the read-side repair, and the raw '2026' would then compare as TEXT instead of as the instant the repair reads it as.
  • The probe promises exactly what an apply rewrites. buildProbeSql gains a fourth count, non_temporal_text — the migration's own WHERE with the guard ANDed on instead of negated — and residual - nonTemporalText is what decides whether the convergence phase runs. The remote side has no other preview surface: SqlDriver.previewDeferredSchemaWork reads deferredSchemaObjects, which only the Knex sync path fills, so in remote mode it answers [] and this probe is the whole plan. residual itself is deliberately not netted down, because that is what holds the mark.
  • A new per-column report field, nonTemporalTextRowsWithheld, plus an operator warning naming the consequence and the remedy. It is null — NOT MEASURED — when no guard was supplied, never 0.
  • No new gate, no new check:* script, and no recommendation for one.

The corrected sentence

The module's contract said rows outside the epoch-recovery band "do not block the canonical mark, because they are fixpoints of the shared repair". That is true only above SQLite's julian-day ceiling, where strftime returns NULL and coalesce hands the value back. Below it — '12', '2026', '86400' — the repair answers, the row is not a fixpoint, and dropping the repair changes what it matches. Both halves are now written with the measurement behind them, and the field's own contract says which of the two a given row is in. The count was not filed under unresolvedEpochTextRows, whose documented meaning is recorded and harmless to the mark; these rows are the opposite, so they got their own name.

Why the entry points are not breaking

The third position of probeRemoteCanonicalColumns, backfillRemoteCanonicalColumn and backfillRemoteCanonicalColumns now accepts RemoteBackfillSqlRules or the pre-existing bare CanonicalSqlFor. The bare arm is fail-closed, not a fallback: with no guard to withhold by, the convergence phase is refused, the column reports error and stays unmarked, and reads stay correct on the repair. So an external caller compiled against 17.4.0 keeps compiling and still cannot write a misreading. A required parameter was considered and rejected: this repo forbids a major changeset during the launch window, and a minor carrying a BREAKING banner would owe an ADR-0087 disposition for which no honest category exists here (runtime-interface-only needs zero in-code references, type-surface-only needs an erased base-side type, no-migration-prescription is closed by the FROM/TO the changeset owes). The union keeps the fix out of that dead end.

The three questions

1. Does the julian misread reach the epoch-text UPDATE? No, and for two independent reasons, either sufficient.

Structural: its SET expression is canonicalSqlFor(kind, 'cast(col as real)'). typeof(cast(X as real)) is always 'real', so the canonical expression takes its integer/real limb — strftime('%Y-%m-%dT%H:%M:%fZ', v/1000.0, 'unixepoch'). The julian limb lives only in the else arm's strftime(..., col), which that statement can never reach, whatever the row holds.

Set-theoretic: its WHERE selects digits-only TEXT in 1e12 <= v < 4.1024448e12, and a bare number is read as a julian day only below 5373484.5. Disjoint (see Q3). ANDing the guard on there would withhold nothing and would falsely imply the two sets meet, so the statement is left alone and the reading is recorded beside it, with a pin.

2. After this change, can a column still be marked canonical while holding a row the shared repair reads as a BC date? No — and the path is named rather than assumed absent. canonical is after.residual === 0 && after.epochInBand === 0. A julian-readable digits-only cell is not equal to what the repair yields for it ('2026' vs '-4707-06-11T12:00:00.000Z'), so col IS NOT canonical holds and the row is counted in residual; it is withheld from the UPDATE, so it stays counted; therefore residual >= 1 and the mark cannot be granted. The early fast-path return is gated on the same probe.residual === 0, so it is not a second door.

Control: the same instrument reports residual = 0 for a digits-only value above the julian ceiling ('1753660800', '999999999999', REMOTE_BACKFILL_EPOCH_MS_MAX + 1) — so the non-zero residual is about the value, not a stuck counter. The 'huge' test asserts that column is still marked, and asserts julianday() is NULL for it, taken from SQLite rather than claimed.

3. Where does the epoch band sit relative to the julian range? There is a gap of nearly six orders of magnitude; the bands neither overlap nor abut.

lower upper
julian-day reading of a bare number 0 5373484.5 (exclusive)
REMOTE_BACKFILL_EPOCH_MS_MIN .. _MAX 1000000000000 4102444800000 (exclusive)

Measured, not derived: julianday('5373484.4') answers 5373484.4, julianday('5373484.5') is NULL, and julianday() is NULL at both ends of the epoch band. The gap 5373484.5 <= v < 1e12 is digits-only text that neither limb touches — genuine coalesce fixpoints, and that is where '1753660800' (epoch seconds) lands. Consequence: the guard cannot withhold a single row the 后果 B recovery would convert, which is pinned end to end by a column holding both an in-band epoch-text row and a withheld julian row.

Verification

Run on the merged tree (origin/main merged, then the branch squashed onto it with the byte-identical tree; final head as pushed).

  • pnpm --filter @objectstack/driver-turso test — 55 files, 1291 tests, 0 failed (1289 before; the 9 new tests, minus the rewritten one, account for it).
  • pnpm --filter @objectstack/driver-turso typecheck — exit 0.
  • pnpm --filter '...@objectstack/driver-turso' typecheck — downstream consumers, scope 21 of 82 workspace projects, exit 0. No in-repo consumer of the three entry points exists outside this package (grepped); the sweep proves the widened third position breaks none of them.
  • pnpm exec turbo run build --filter='./packages/*' --filter='./packages/*/*' — 72/72, exit 0.
  • pnpm lint (repo-wide eslint . --no-inline-config) — exit 0. Not narrowed; the whole scan ran.
  • Gate families: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 61 commands. All 61 run on the merged, built tree, each exit code captured before any pipe; 61 run, 0 NOT-MEASURED, 0 UNRUN, reconciled with --ran carrying every exit code (a derived zero, not a claimed one). Three families first answered exit 3 PREREQUISITE NOT MET (check:dual-build-cjs-loads, check:lean-entry-closure, check:type-check-debt) and were discharged by building the closure they named, never by declaring them inapplicable.
  • check:doc-authoring caught a real finding on the first pass — the refusal string carried a tracker id an operator cannot resolve. The id moved to an adjacent comment and the gate is green.

Ablation

One anchor, resolveBackfillSql's nonTemporalText: sql.nonTemporalText, replaced with a predicate that matches nothing — the single funnel both the probe count and the UPDATE's WHERE read, so the mutated module behaves exactly as the pre-fix one. Driven through scripts/ablation-replace.mjs, which proves the write against the disk:

anchor   "nonTemporalText: sql.nonTemporalText"  x1 -> x0
replace  "nonTemporalText: () => '(0)'"          x0 -> x1
blob     dcbb875a6bfb7281ac7cd561532752d6e72f117b -> 1dd0ca3cfab510d38ace81d4de30bea0975e1995

Result: 5 failed | 24 passed (29). The subject is resolved from source by a relative in-package import, so no build sits between the mutation and the run.

Red, each for the right reason: the datetime row became -4707-06-11T12:00:00.000Z; the time row became 12:00:00; 'now' became the run's own wall clock; the plan's residual read 3 instead of 4; and the julian row in the mixed column was rewritten.

Deliberate positive controls that stayed green, so the suite cannot pass vacuously: the above-the-ceiling fixpoint row is still marked canonical; the not-over-broad case (naive, offset, canonical and junk rows) still converges and marks; the band-disjointness measurement is unmoved; both fail-closed cases are unmoved (they take the bare arm, which the mutation does not reach); and every #5770 batching, resumability, failure and identifier-safety test is unmoved.

Restore proven, not assumed: blob back to dcbb875a6bfb, git diff HEAD empty, and whole-tree git status --porcelain empty.

Changeset

.changeset/6009-turso-remote-julian-guard.md, @objectstack/driver-turso: minor. Measured rather than assumed: the package is not private, its files[] ships dist, and the changed bytes reach it — nonTemporalTextRowsWithheld appears 7x in dist/index.js, 7x in dist/index.mjs and 2x in dist/index.d.ts, RemoteBackfillSqlRules 4x in the .d.ts, and the refusal string once in dist/index.js, against a negative control that matches nothing. minor rather than patch because new exported types and a new report field land; no BREAKING banner, because nothing breaks.

Acceptance notes

  • unresolvedEpochTextRows and nonTemporalTextRowsWithheld deliberately overlap and neither is a subset of the other: the guard also matches shapes the digits-only count never sees (a leading + or space, '2026e0', a digits-only BLOB — all measured). Subtracting one from the other to make the older field's sentence uniformly true would have produced a count that can go negative; the fields' contracts say which rows are in which half instead.
  • The convergence phase still issues one statement that matches zero rows when every residual row is withheld, exactly as the local twin does. Noted, not filed: one wasted statement on a column that cannot converge, on a path that already costs a probe.

Generated by Claude Code

…n misreading over a stored value

SQLite's time-value grammar accepts a bare number as a JULIAN DAY and `now` as
the wall clock, so for those two shapes `strftime` answers confidently instead
of returning NULL and the `coalesce(strftime(...), col)` that is supposed to
preserve unparseable values never fires. The REMOTE backfill's convergence
`UPDATE` therefore wrote that answer over the stored bytes, irrecoverably.

The driver now hands its own `sqliteNonTemporalTextSql` across the module
boundary alongside its canonical expression; rows the predicate matches are
withheld from the `UPDATE`, counted, and — because a withheld row is not a
fixpoint of the shared repair and so stays inside the probe's `residual` — they
also block the canonical mark, which is what keeps every query answer identical.

The third position of the three exported entry points now accepts either the
rules object or the pre-existing bare canonical expression; the bare arm is
fail-closed (the convergence phase is refused rather than run unguarded), so no
caller's compile breaks and none of them can write a misreading either.

Claude-Session: https://claude.ai/code/session_01NcPSwnmJHczmTu6FG7NMjE
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/l documentation Improvements or additions to documentation tests tooling labels Sep 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/driver-turso, touching 19 documentable anchor(s).

2 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/data-modeling/drivers.mdx (via TursoDriver (symbol, a top-level class))
  • content/docs/plugins/packages.mdx (via TursoDriver (symbol, a top-level class))
What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 60 of 215 client-bound route-ledger rows — the other 155 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 155: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 100 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json a251aaa19bf94741aa24205675eb942d5e8ae370 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from cac8cce980be2230fd57ffa63f1cedc389cf0c71 — the merge of head 0801c9f28d0bef55dd20487629ab8e5419ff592a into base a251aaa19bf94741aa24205675eb942d5e8ae370, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin cac8cce980be2230fd57ffa63f1cedc389cf0c71 && git checkout cac8cce980be2230fd57ffa63f1cedc389cf0c71
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a251aaa19bf94741aa24205675eb942d5e8ae370 0801c9f28d0bef55dd20487629ab8e5419ff592a && git checkout -B drift-repro a251aaa19bf94741aa24205675eb942d5e8ae370 && git merge --no-ff 0801c9f28d0bef55dd20487629ab8e5419ff592a

node scripts/docs-audit/affected-docs.mjs --json a251aaa19bf94741aa24205675eb942d5e8ae370

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs a251aaa19bf94741aa24205675eb942d5e8ae370 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@huangyiirene
huangyiirene marked this pull request as ready for review September 22, 2026 10:59
@huangyiirene
huangyiirene added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit e2c55ed Sep 22, 2026
36 checks passed
@huangyiirene
huangyiirene deleted the claude/issue-6009-turso-julian-guard branch September 22, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

drivers(sql): 共享 canonical 修复表达式把「裸数字文本」当儒略日读,backfill 会把这个误读写进盘里(观察项)

2 participants