Repository navigation
fix(driver-turso): the remote canonical backfill never writes a julian misreading over a stored value - #19719
Conversation
…n misreading over a stored value SQLite's time-value grammar accepts a bare number as a JULIAN DAY and `now` as the wall clock, so for those two shapes `strftime` answers confidently instead of returning NULL and the `coalesce(strftime(...), col)` that is supposed to preserve unparseable values never fires. The REMOTE backfill's convergence `UPDATE` therefore wrote that answer over the stored bytes, irrecoverably. The driver now hands its own `sqliteNonTemporalTextSql` across the module boundary alongside its canonical expression; rows the predicate matches are withheld from the `UPDATE`, counted, and — because a withheld row is not a fixpoint of the shared repair and so stays inside the probe's `residual` — they also block the canonical mark, which is what keeps every query answer identical. The third position of the three exported entry points now accepts either the rules object or the pre-existing bare canonical expression; the bare arm is fail-closed (the convergence phase is refused rather than run unguarded), so no caller's compile breaks and none of them can write a misreading either. Claude-Session: https://claude.ai/code/session_01NcPSwnmJHczmTu6FG7NMjE Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 1 package(s): 2 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
What this run could not see
Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin cac8cce980be2230fd57ffa63f1cedc389cf0c71 && git checkout cac8cce980be2230fd57ffa63f1cedc389cf0c71
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a251aaa19bf94741aa24205675eb942d5e8ae370 0801c9f28d0bef55dd20487629ab8e5419ff592a && git checkout -B drift-repro a251aaa19bf94741aa24205675eb942d5e8ae370 && git merge --no-ff 0801c9f28d0bef55dd20487629ab8e5419ff592a
node scripts/docs-audit/affected-docs.mjs --json a251aaa19bf94741aa24205675eb942d5e8ae370
|
Fixes #6009
Clause-②: no
The sequel to PR #19688, which fixed the
driver-sql(Knex) half of this card at squash40626bdca6. The REMOTE path is a separate module that builds its own statements and did not inherit the guard.The defect
RemoteTransport.mapFieldTypeToSQLdeclares every temporal columnTEXT, so aField.datetime/Field.timecolumn can hold a digits-only string such as'2026','86400','12'— or the keyword'now'. SQLite's time-value grammar accepts a bare number as a JULIAN DAY andnowas the wall clock, so for those two shapesstrftimeanswers confidently rather than returning NULL, and thecoalesce(strftime(...), col)that is supposed to preserve unparseable values never fires. The convergenceUPDATEinbackfillRemoteCanonicalColumnwrote that answer over the stored bytes, and no later run can get them back.Measured here on better-sqlite3 13.0.3 / SQLite 3.53.4, column declared
TEXT:In a
Field.timecolumn the same'2026'becomes'12:00:00'— a wall clock nobody wrote. Both are pinned.What changed
All of it inside
packages/drivers/driver-turso/src/. Nopackages/drivers/driver-sql/file is touched, and neithersqliteCanonicalDatetimeSqlnorsqliteCanonicalTimeSqlmoves — the 2026-08-03 cloud#1005 ruling that refused a heuristic in the shared read expression is untouched, and #19688's pins that those expressions still misread a bare number stay green.TursoDriverhands its ownsqliteNonTemporalTextSqlacross the module boundary, beside the canonical expression it already hands over, as{ canonical, nonTemporalText }. The predicate isprotectedonSqlDriverand stays that way; nothing is copied, so remote and local cannot drift about which bytes are safe to overwrite.UPDATEgainsand not (guard)— the local twin's exclusion, same construction.col IS NOT canonical, so it is already inside the probe'sresidual, and the mark already requiresresidual === 0. Withholding alone would not do it — a marked column drops the read-side repair, and the raw'2026'would then compare as TEXT instead of as the instant the repair reads it as.buildProbeSqlgains a fourth count,non_temporal_text— the migration's ownWHEREwith the guard ANDed on instead of negated — andresidual - nonTemporalTextis what decides whether the convergence phase runs. The remote side has no other preview surface:SqlDriver.previewDeferredSchemaWorkreadsdeferredSchemaObjects, which only the Knex sync path fills, so in remote mode it answers[]and this probe is the whole plan.residualitself is deliberately not netted down, because that is what holds the mark.nonTemporalTextRowsWithheld, plus an operator warning naming the consequence and the remedy. It isnull— NOT MEASURED — when no guard was supplied, never0.check:*script, and no recommendation for one.The corrected sentence
The module's contract said rows outside the epoch-recovery band "do not block the canonical mark, because they are fixpoints of the shared repair". That is true only above SQLite's julian-day ceiling, where
strftimereturns NULL andcoalescehands the value back. Below it —'12','2026','86400'— the repair answers, the row is not a fixpoint, and dropping the repair changes what it matches. Both halves are now written with the measurement behind them, and the field's own contract says which of the two a given row is in. The count was not filed underunresolvedEpochTextRows, whose documented meaning is recorded and harmless to the mark; these rows are the opposite, so they got their own name.Why the entry points are not breaking
The third position of
probeRemoteCanonicalColumns,backfillRemoteCanonicalColumnandbackfillRemoteCanonicalColumnsnow acceptsRemoteBackfillSqlRulesor the pre-existing bareCanonicalSqlFor. The bare arm is fail-closed, not a fallback: with no guard to withhold by, the convergence phase is refused, the column reportserrorand stays unmarked, and reads stay correct on the repair. So an external caller compiled against 17.4.0 keeps compiling and still cannot write a misreading. A required parameter was considered and rejected: this repo forbids amajorchangeset during the launch window, and aminorcarrying a BREAKING banner would owe an ADR-0087 disposition for which no honest category exists here (runtime-interface-onlyneeds zero in-code references,type-surface-onlyneeds an erased base-side type,no-migration-prescriptionis closed by the FROM/TO the changeset owes). The union keeps the fix out of that dead end.The three questions
1. Does the julian misread reach the epoch-text
UPDATE? No, and for two independent reasons, either sufficient.Structural: its SET expression is
canonicalSqlFor(kind, 'cast(col as real)').typeof(cast(X as real))is always'real', so the canonical expression takes its integer/real limb —strftime('%Y-%m-%dT%H:%M:%fZ', v/1000.0, 'unixepoch'). The julian limb lives only in theelsearm'sstrftime(..., col), which that statement can never reach, whatever the row holds.Set-theoretic: its
WHEREselects digits-only TEXT in1e12 <= v < 4.1024448e12, and a bare number is read as a julian day only below5373484.5. Disjoint (see Q3). ANDing the guard on there would withhold nothing and would falsely imply the two sets meet, so the statement is left alone and the reading is recorded beside it, with a pin.2. After this change, can a column still be marked canonical while holding a row the shared repair reads as a BC date? No — and the path is named rather than assumed absent.
canonicalisafter.residual === 0 && after.epochInBand === 0. A julian-readable digits-only cell is not equal to what the repair yields for it ('2026'vs'-4707-06-11T12:00:00.000Z'), socol IS NOT canonicalholds and the row is counted inresidual; it is withheld from theUPDATE, so it stays counted; thereforeresidual >= 1and the mark cannot be granted. The early fast-path return is gated on the sameprobe.residual === 0, so it is not a second door.Control: the same instrument reports
residual = 0for a digits-only value above the julian ceiling ('1753660800','999999999999',REMOTE_BACKFILL_EPOCH_MS_MAX + 1) — so the non-zero residual is about the value, not a stuck counter. The'huge'test asserts that column is still marked, and assertsjulianday()is NULL for it, taken from SQLite rather than claimed.3. Where does the epoch band sit relative to the julian range? There is a gap of nearly six orders of magnitude; the bands neither overlap nor abut.
05373484.5(exclusive)REMOTE_BACKFILL_EPOCH_MS_MIN.._MAX10000000000004102444800000(exclusive)Measured, not derived:
julianday('5373484.4')answers5373484.4,julianday('5373484.5')is NULL, andjulianday()is NULL at both ends of the epoch band. The gap5373484.5 <= v < 1e12is digits-only text that neither limb touches — genuinecoalescefixpoints, and that is where'1753660800'(epoch seconds) lands. Consequence: the guard cannot withhold a single row the 后果 B recovery would convert, which is pinned end to end by a column holding both an in-band epoch-text row and a withheld julian row.Verification
Run on the merged tree (
origin/mainmerged, then the branch squashed onto it with the byte-identical tree; final head as pushed).pnpm --filter @objectstack/driver-turso test— 55 files, 1291 tests, 0 failed (1289 before; the 9 new tests, minus the rewritten one, account for it).pnpm --filter @objectstack/driver-turso typecheck— exit 0.pnpm --filter '...@objectstack/driver-turso' typecheck— downstream consumers, scope 21 of 82 workspace projects, exit 0. No in-repo consumer of the three entry points exists outside this package (grepped); the sweep proves the widened third position breaks none of them.pnpm exec turbo run build --filter='./packages/*' --filter='./packages/*/*'— 72/72, exit 0.pnpm lint(repo-wideeslint . --no-inline-config) — exit 0. Not narrowed; the whole scan ran.node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderived 61 commands. All 61 run on the merged, built tree, each exit code captured before any pipe; 61 run, 0 NOT-MEASURED, 0 UNRUN, reconciled with--rancarrying every exit code (a derived zero, not a claimed one). Three families first answered exit 3 PREREQUISITE NOT MET (check:dual-build-cjs-loads,check:lean-entry-closure,check:type-check-debt) and were discharged by building the closure they named, never by declaring them inapplicable.check:doc-authoringcaught a real finding on the first pass — the refusal string carried a tracker id an operator cannot resolve. The id moved to an adjacent comment and the gate is green.Ablation
One anchor,
resolveBackfillSql'snonTemporalText: sql.nonTemporalText, replaced with a predicate that matches nothing — the single funnel both the probe count and theUPDATE'sWHEREread, so the mutated module behaves exactly as the pre-fix one. Driven throughscripts/ablation-replace.mjs, which proves the write against the disk:Result: 5 failed | 24 passed (29). The subject is resolved from source by a relative in-package import, so no build sits between the mutation and the run.
Red, each for the right reason: the datetime row became
-4707-06-11T12:00:00.000Z; the time row became12:00:00;'now'became the run's own wall clock; the plan'sresidualread 3 instead of 4; and the julian row in the mixed column was rewritten.Deliberate positive controls that stayed green, so the suite cannot pass vacuously: the above-the-ceiling fixpoint row is still marked canonical; the not-over-broad case (naive, offset, canonical and junk rows) still converges and marks; the band-disjointness measurement is unmoved; both fail-closed cases are unmoved (they take the bare arm, which the mutation does not reach); and every #5770 batching, resumability, failure and identifier-safety test is unmoved.
Restore proven, not assumed: blob back to
dcbb875a6bfb,git diff HEADempty, and whole-treegit status --porcelainempty.Changeset
.changeset/6009-turso-remote-julian-guard.md,@objectstack/driver-turso: minor. Measured rather than assumed: the package is not private, itsfiles[]shipsdist, and the changed bytes reach it —nonTemporalTextRowsWithheldappears 7x indist/index.js, 7x indist/index.mjsand 2x indist/index.d.ts,RemoteBackfillSqlRules4x in the.d.ts, and the refusal string once indist/index.js, against a negative control that matches nothing.minorrather thanpatchbecause new exported types and a new report field land; no BREAKING banner, because nothing breaks.Acceptance notes
unresolvedEpochTextRowsandnonTemporalTextRowsWithhelddeliberately overlap and neither is a subset of the other: the guard also matches shapes the digits-only count never sees (a leading+or space,'2026e0', a digits-only BLOB — all measured). Subtracting one from the other to make the older field's sentence uniformly true would have produced a count that can go negative; the fields' contracts say which rows are in which half instead.Generated by Claude Code