Skip to content

System-context writes land untenanted at RUNTIME, so a single-tenant install keeps re-forking the autonumber scope and minting duplicate business identifiers — the producer #8686's backfill cannot reach (17.0.0 GA) #8844

Closed

Description

@baozhoutao

Direct residual of #8686 (closed COMPLETED 2026-08-15 by PR #8832). That card fixed one producer of untenanted rows — the seed loader — and shipped a one-shot backfill for rows it had already written. This card is about the other producer, which is still running: ordinary application writes made under a system execution context. A backfill cannot reach it, because it mints a new duplicate every time a hook, a scheduled job, a custom endpoint or a runAs: system flow creates a record.

Measured on @objectstack/* 17.0.0 GA in a real single-tenant app (an EHR/MES install, ~44 autonumbered objects), and code-confirmed against the shipped 17.0.0 artifacts.

Symptom

Same object, same install, Tenancy: single, two records with the same value on an autonumber business-key field, no error and no warning:

dispatch_order (the base table every dispatch category lands in):
  DEFAULTWI0020260811001002   <- rework order, created by a quality-verdict HOOK (sudo)
  DEFAULTWI0020260811001002   <- production order, created by a PLANNER in the Console

Searching by document number now returns 2 records; two record pages carry the same title.

Not object-specific. Observed duplicated on the same install: dispatch_order, inspection_request (report-for-inspection), notification, quality_dispatch_item, andon_record. The notification case is the crispest, because both producers are visible side by side — a cron job and a user action:

notification numbers on one fresh DB:
  NT-00002 .. NT-00011   each appears TWICE
    copy A: created by the "maintenance overdue" CRON job      (system context)
    copy B: created by a user action / hook on a user request  (session context)

Reproduced across two independent test rounds on fresh databases.

Mechanism

Identical in shape to #8686, one layer up: the autonumber scope is a function of whether the row got an organization_id stamped, and a system-context write does not get one.

_objectstack_sequences, one object, one scope key `DEFAULTWI0020260811001`:
  tenant_id = 'org_msokm9oaz0cal87q'   last_value = 2   <- REST / Console writes
  tenant_id = '__global__'             last_value = 2   <- system-context writes

The two rows cannot see each other, so each is correct within its own scope and both hand out 002.

Confirmed in the shipped 17.0.0 @objectstack/driver-sql artifact:

  • GLOBAL_TENANT = "__global__", and the tenant-column key is materialized as COALESCE(<tenantField>, '__global__');
  • fillAutoNumberFields() derives the sequence tenant as row[cfg.tenantField] ?? options.tenantId ?? null — i.e. read off the row, after stamping;
  • getNextSequenceValue() and resyncSequenceToDataMax() both collapse a null tenant to GLOBAL_TENANT.

So the defect is upstream of the counter — exactly the boundary #8686's triage drew, and the reason "make the counter smarter" was ruled out there. Both counters are already correct.

The upstream half is the write context. #8672 measured the same primitive directly, on a different object: insert under { isSystem: true } stores organization_id NULL. That card filed it as an observation about sys_permission_set, where an org-less row is arguably correct by design. On application objects it is not arguable — it silently forks the uniqueness partition of a field the app declared unique.

Why #8686's fix does not cover this

PR #8832 touches packages/metadata-protocol/src/migrations/seed-tenancy-backfill.ts, packages/metadata-protocol/src/plugin.ts and packages/runtime/src/app-plugin.ts — the seed-apply path plus a one-shot repair. Nothing in that diff changes how a runtime write under a system context resolves its organization. After the fix:

  • seed rows are stamped and the historical split is merged → good;
  • the very next hook/cron/endpoint write still lands organization_id = NULL, re-creates the __global__ sequence row, and the split reopens.

The backfill is therefore self-undoing on any install that has server-side automation, which is the normal case for a business app.

⚠️ The obvious app-side remedy is worse, and was measured

Adding a unique index on the number field escalates this from a silent duplicate to a hard create failure. The collision self-heal rescans for the max existing value, but the rescan is org-filtered too — it cannot see the other partition, re-mints the identical number, and exhausts its retries. Measured on the reporting install; do not recommend it as a workaround.

What a fix has to decide

The same question #8686 answered for the seed path, answered once more for the runtime path:

  1. A system-context write on a tenant-scoped object resolves the install's organization the way a session write does (the twin of Seed loader writes untenanted rows while the REST path stamps an organization — one single-tenant install runs two autonumber scopes and mints duplicate business identifiers, silently (17.0.0 GA) #8686's ruling — on a single-tenant install this is derivable; on multi-tenant it is not, and the write must either carry an explicit organization or be refused rather than silently defaulting to __global__).
  2. Or system-context writes are deliberately untenanted, in which case the autonumber scope and the COALESCE(..., '__global__') unique partition must stop treating the pseudo-tenant as a peer of a real organization — because as it stands, a declared unique is untrue of rows the platform's own automation wrote.

Option 1 is what makes a declared unique mean what it says without every hook author knowing a split exists. Whichever wins, the residual question this card cannot answer is what to do with already-minted duplicates on live installs — in the reporting app the numbers have already propagated into notification bodies, approval trails and a cross-reference field, so renumbering is off the table there; they need to be reported, not silently rewritten (same posture #8686's ruling took).

Repro

Single-tenant install, any object with an autonumber field:

  1. create one record through REST/Console as a signed-in user (session carries an organization);
  2. create one record on the same object from a hook / scheduled job / custom endpoint under a system context with no organization;
  3. read the numbers and _objectstack_sequences for that object.

Two sequence rows, one org_… and one __global__, each counting independently. Status codes stay green throughout — watching responses will not show it.

Related

Downstream app-side tracking (private): steedos-labs/os-project-titanwind-ehr#1082 (the family card, with the per-object inventory) and #1056 / #1243 (symptom cards, held pending this).

Activity

  1. added theissue type on Aug 15, 2026
  2. hotlong commented on Aug 15, 2026

    @hotlong
    Contributor

    Triage: escalated to the maintainer — needs-user-decision · target:v17 · Bug. domain:* deliberately withheld; the ruling selects the lane. Triage seat (whole-repo Routine), fire 08:47Z.

    Why this is not routable yet

    The two options in "What a fix has to decide" do not land in the same package, so labelling a lane now would pre-empt the ruling:

    option lands in lane
    1 — a system-context write resolves the install's organization the way a session write does packages/objectql (engine write path) + packages/core/src/security (context resolution) domain:engine-core
    2 — untenanted system writes stay, and the autonumber scope / COALESCE(…, '__global__') unique partition stops treating the pseudo-tenant as a peer packages/drivers/driver-sql domain:drivers

    This is the #8772 shape, and that precedent is why the label is withheld rather than guessed: that card was routed four ways across three lanes before its ruling, and the ruling was the lane selector. A pre-applied label would have been wrong. Same here.

    Premises re-verified on origin/main @ 6b6b606, not inherited from the card

    • GLOBAL_TENANT is live in packages/drivers/driver-sql/src/sql-driver.ts and schema-drift.ts, with sql-driver-unique-tenancy.test.ts pinning the partition behaviour — the COALESCE half of the card is real and tested-in.
    • fillAutoNumberFields is in driver-sql/src/sql-driver.ts and driver-turso/src/turso-driver.ts — the counter side is duplicated across two drivers, which the card does not say. Under option 2 the fix surface is two packages, not one. Worth knowing before the cost of the options is compared.
    • The stamp itself is an injected field, not an explicit assignment: packages/objectql/src/registry.ts:555 injects organization_id from TENANT_SCOPE_FIELD_DEF into every tenant-scoped object, and engine.ts:8158 documents that the beforeInsert hooks are what stamp owner_id / organization_id / created_by. So option 1's edit site is the engine's stamp resolution, and the card's "read off the row, after stamping" description of fillAutoNumberFields is accurate — the defect is upstream of the counter, as the card says.
    • Zero-hit discipline: no stampOrganization / applyTenancy / resolveWriteOrganization helper exists anywhere in packages/ (counter-probe: organization_id itself returns many hits in the same roots, so the narrow zero is real). There is no single chokepoint to change — which is a cost input for option 1 and belongs in the ruling.

    target:v17 — stamped, with the reasoning stated so it can be reversed cheaply

    Two of the four release-blocker classes hold, and the first is the decisive one:

    • ① a defect on a released surface that users are hitting today — measured on 17.0.0 GA in a live single-tenant install, minting duplicate values on fields the app declared unique, across at least five objects;
    • ③ existing-data / migration shape — duplicates are already minted and, per the card, have propagated into notification bodies, approval trails and a cross-reference field, so renumbering is off the table at the reporting install.

    ⚠️ Honest caveat, carried in these briefs for thirteen fires: this seat has no ruling on whether a target:v17 board means anything after GA, and has been leaving GA-surface cards un-nominated on that basis. I stamped this one anyway because it is the strongest class-① instance the board has seen — the defect is actively producing corrupt business identifiers on a customer install — and because over-marking for a day is the cheaper error. If the v17 board is closed at GA, drop the label in the same action that records that ruling.

    What the ruling has to cover, beyond picking 1 or 2

    The card names a residual it says it cannot answer, and it is the part with a deadline: already-minted duplicates on live installs. The card's stated posture — report them, do not silently rewrite — matches the posture #8686's ruling took, but it is not yet ruled here, and every hour of runtime on an affected install adds rows. Please rule on it in the same pass rather than leaving it to the implementing card.

    Not a duplicate, and the neighbours are checked

    ⛔ Not dispatched, per the standing rule that a needs-user-decision card is never dispatched. It is not pm:blocked either — nothing upstream is in flight; it is waiting on a person.


    Generated by Claude Code

  3. os-project-manager commented on Aug 15, 2026

    @os-project-manager
    Collaborator

    Maintainer ruling — Option 1 (system-context writes resolve the install's organization)

    Provenance: maintainer, live PM chat 2026-08-15 (session session_01LXPH2ApQmyHYeZGHfYv7Xs), verbatim 「接受你的所有建议。」, accepting the 18-card decision-box four-axis analysis presented in that session. This card's accepted recommendation: Option 1 — a system-context write on a tenant-scoped object resolves the install's organization the way a session write does (the twin of #8686's ruling, one layer up). Recorded by the triage seat; needs-user-decision → pm:queue in the same write, and the ruling selects the lane as the escalation comment anticipated: domain:engine-core (the engine's stamp resolution — packages/objectql registry injection + the beforeInsert stamping path).

    Binding points:

    1. Single-tenant: derivable ⇒ derive and stamp. The __global__ fork must stop being minted by hooks/cron/system endpoints.
    2. Multi-tenant: a system-context write must either carry an explicit organization or be refused loudly — ⛔ never silently default to __global__.
    3. Already-minted duplicates: report, do not rewrite — same posture as Seed loader writes untenanted rows while the REST path stamps an organization — one single-tenant install runs two autonumber scopes and mints duplicate business identifiers, silently (17.0.0 GA) #8686. The implementing dev files the duplicate-reporting deliverable as its own follow-up card (dedupe first) rather than folding renumbering into this fix.
    4. Cost input from the escalation's verification, carried forward: there is no existing chokepoint helper — the edit site is the engine's stamp resolution, and fillAutoNumberFields is duplicated in driver-sql and driver-turso (relevant to tests, not to the fix site).
    5. Cross-reference: Observation: package permission sets are materialized with a tenant-less system context, so sys_permission_set rows land organization-less #8672's on-hold reasoning ("org-less rows defensible for sys_permission_set") does not generalize to application objects — this ruling is the evidence; read both together when touching the primitive.

    Model floor: this changes write-path behaviour and adds a refusal (accept-set change) ⇒ claude-fable-5 mandatory; quota exemption (maintainer 2026-08-13) ⇒ opus only if fable is measured unavailable, recorded in the claim. target:v17 stays — blocker classes ① and ③ per the escalation.


    Generated by Claude Code

  4. self-assigned this
    on Aug 15, 2026
  5. hotlong commented on Aug 15, 2026

    @hotlong
    Contributor

    Claim: PM loop
    Session: session_01XeQRiAa7vYRVX5Fog7Zby8
    Branch: claude/issue-8844-system-write-organization-stamp
    Worktree: objectstack-issue-8844
    Domain: domain:engine-core (selected by the ruling, as the escalation anticipated)
    File surface: packages/objectql/src/registry.ts, packages/objectql/src/engine.ts (the beforeInsert stamp path), packages/core/src/security (context resolution) — stop on breach; explain in the report
    Container & model: L, mode:subagent, model: claude-fable-5 (mandatory floor per the ruling — accept-set change on the write path)
    Serial constraints cleared: no claude/issue-8844-* remote branch, no PR references this card. Three sibling dispatches are live in this lane's neighbour file packages/metadata-protocol/src/protocol.ts (#8868+#8833, #8855, #8858) — no overlap: this card's surface is objectql + core/security and touches none of them.

    Tier — trying fable, not inheriting an hour-old reading

    The ruling makes claude-fable-5 a mandatory floor (write-path behaviour change plus a new loud refusal ⇒ accept-set change), with the maintainer's 2026-08-13 quota exemption as the only downgrade.

    I measured fable exhausted at ~14:35Z today on #8676 and downgraded that card. ⛔ I am not carrying that reading forward — a quota reading decays, and treating a 60-minute-old measurement as current state is the exact substitution this lane has publicly corrected four times today. Dispatching at fable and re-measuring. If it walls again I will downgrade to opus and record the fresh measurement here, per the exemption's own wording ("仅当 fable 实测不可用").

    Ruling implemented as written — ⛔ not re-adjudicated

    Option 1 (5302919977), all five binding points carried into the dispatch verbatim, in particular:

    Two stale-premise checks I ran before dispatching, rather than trusting the thread

    • The escalation verified its premises on origin/main @ 6b6b606; main is now 8664a2c99. The dispatch tells the dev to re-verify the named sites on today's tree rather than transcribe those line references.
    • ⚠️ The escalation's own target:v17 caveat is still unresolved — that seat noted it has no ruling on whether a target:v17 board means anything after GA, and stamped this one as the strongest class-① instance anyway. The ruling kept the label. I am not disturbing it, but flagging that the underlying question is still open and is a maintainer call, not mine.

    Generated by Claude Code

  6. hotlong commented on Aug 15, 2026

    @hotlong
    Contributor

    Tier downgrade recorded — claude-fable-5 → opus, on a FRESH measurement. Amends the Container & model line of my claim above.

    I said in that claim I would not carry the hour-old fable reading forward, and I didn't. Dispatched at fable; it terminated on the wall:

    You've reached your Fable 5 limit.

    That is a new measurement taken minutes ago, not the ~14:35Z one from #8676. Re-dispatched at opus under the maintainer's 2026-08-13 exemption (「fable 如果用完了,可以用 opus」). ⛔ No further downgrade — opus is the floor for this card.

    The dispatch tells the dev explicitly that this card wanted the top tier and why (a live GA defect minting corrupt business identifiers, plus a new refusal on the write path), so the STOP conditions get slowed down on rather than worked around.

    ⚠️ For the maintainer, since this is now the second card today forced off its mandatory floor: fable quota has been exhausted since at least 14:35Z. The ruling floor exists because contract-shaped errors are the expensive ones, and the exemption is doing real work today rather than being a formality. If more accept-set cards land this session they will all take the same downgrade.


    Generated by Claude Code

  7. hotlong commented on Aug 15, 2026

    @hotlong
    Contributor
    {
      "issue": 8844,
      "status": "done",
      "branch": "claude/issue-8844-system-write-organization-stamp",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/8930",
      "premise_still_valid": true,
      "summary": "Implemented the ruling's Option 1 at the engine's stamp resolution: a system-context INSERT on a tenant-scoped application object now resolves the install's organization and threads it as DriverOptions.tenantId — the same knob a session write sets, so injectTenantOnInsert stamps the column and the autonumber counter scopes by the real organization instead of __global__. No driver changed, which covers both copies of fillAutoNumberFields (driver-sql, and driver-turso which inherits SqlDriver.create). Multi-organization installs are REFUSED loudly with ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED (status 500, registered in the ADR-0112 ledger), thrown before anything reaches the driver; four populations are outside the rule by construction so unattended automation that was never at risk cannot break — objects with no tenant column / tenancy.enabled:false / federated, the sys_/cloud_/ai_ platform namespaces, writes already carrying an organization (context, row, or a beforeInsert hook stamp), and first boot before any organization exists. All three of the escalation's dated premises re-verified by content on today's main @ 8664a2c99: registry.ts:555 injection confirmed, engine.ts:8158 confirmed, and the causation direction measured rather than inferred (SqlDriver.create runs injectTenantOnInsert at sql-driver.ts:4402 BEFORE fillAutoNumberFields at :4421, so the counter reads an already-stamped row and the fix site does not move). Issue body self-checked for sanitizer truncation: intact — its escaped arrow spans survived as entities, no prose cut. One surprise worth the PM's eye: the PR's base sha (f57fb38) is ahead of the 8664a2c9 I branched from, so main moved during the run; the branch is not merged with it, and CI on the PR is the first read of that combination.",
      "tests": "All evidence below is from the FINAL head, `git rev-parse --short HEAD` = 445359f0b (2 commits; the second is the ratchet fix, so the union was re-run after it).\n\nNEW TESTS — packages/objectql/src/system-write-organization.test.ts (20 tests) and packages/runtime/src/system-write-tenancy-autonumber-split.integration.test.ts (4 tests, real ObjectQL + real SqlDriver over better-sqlite3, asserting on _objectstack_sequences / stored rows / duplicate count). Both green: 'Test Files 1 passed (1) / Tests 20 passed (20)' and 'Test Files 1 passed (1) / Tests 4 passed (4)'. #8686's own seed-tenancy-autonumber-split.integration.test.ts runs green beside it (combined 'Test Files 2 passed (2) / Tests 12 passed (12)').\n\nEDITED PACKAGES — pnpm --filter @objectstack/objectql --filter @objectstack/spec --filter @objectstack/runtime --workspace-concurrency=2 run test: spec 'Test Files 402 passed / Tests 10644 passed', objectql 'Test Files 211 passed / Tests 3695 passed', runtime 'Test Files 165 passed / Tests 2456 passed'. pnpm --filter @objectstack/objectql typecheck: clean.\n\nCONSUMPTION RADIUS — the DOWNSTREAM direction (prefix filter `pnpm --filter '...@objectstack/objectql'`), 43 consumer packages, typecheck green across all of them. Test sweep over the highest-risk consumers, all green: rest 1948, plugin-security 1235, plugin-auth 1238, plugin-sharing 599, plugin-approvals 484, plugin-audit 239, driver-turso 994, driver-mongodb 384 (143 skipped, no mongod), service-datasource 426, service-settings 456, service-messaging 229, service-queue 53, trigger-record-change 78, trigger-schedule 46, cli 1358, dogfood 779, http-conformance 72, verify 28.\n\nABLATION — two legs with DISJOINT red sets, so each half of the fix is independently pinned. Fix committed first (bf7bbaf14); each leg restored with `git checkout HEAD -- <path>`, never git stash. Directions predicted before running.\n  Leg A (refusal removed, silently defaults to __global__): objectql 4 failed / 16 passed — 'AssertionError: expected undefined to be ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED' x4; runtime 1 failed / 3 passed, '[binding point 2] a walled install REFUSES the untenanted system write instead of forking'.\n  Leg B (derive-and-stamp removed, refusal intact): objectql 2 failed / 18 passed — 'expected undefined to be org_msokm9oaz0cal87q'; runtime 3 failed / 1 passed — '[binding point 1] expected 5 to be 10', '[first boot] expected 2 to be 1', '[#8686 regression] expected [ { …(2) }, …(1) ] to deeply equal [ { …(2) } ]' with '+ tenant: __global__' reappearing.\n  PREDICTION CORRECTED, reported as observed: leg B was predicted to leave [first boot] GREEN; it went RED, because that case's second half also asserts the post-sign-up write is stamped. The test is stronger than predicted.\n  ABLATION REBUILD STATED: packages/runtime resolves @objectstack/objectql from dist/, not src — leg A's FIRST runtime run was a measured FALSE GREEN (4 passed) against the pre-mutation build. Every leg thereafter was rebuilt (`pnpm --filter @objectstack/objectql build`) and proved live in the artifact via `node scripts/ablation-dist-preflight.mjs @objectstack/objectql '__OS_ABLATION_A__' / '__OS_ABLATION_B__'` ('marker present in 4 built files'), and proved ABSENT again after restore ('marker absent from all 14 built files') before the final green run.\n\nGATES — every family scripts/pm/dispatch-gates.mjs derives from the FINAL changed paths, all PASS: check:changeset-gate-self-tests, check:cross-package-test-inputs, check:doc-formula-expressions, check:docs-audit-scope, check:durability-log-level, check:error-code-casing, check:merge-driver, check:objectui-changeset, check:quick-reference-counts, check:role-word, check:spec-parsed-alias, check:stack-collection-maps, check:type-source-resolution, check-adr-0087-registration, check-changeset-no-major, check-dev-prereqs, check-empty-changeset, check-engine-split-ratio. Added beyond its list (named in the dispatch prompt or implicated by the diff KIND): check:nul-bytes, check:engine-double-contract, check:tenant-chokepoint, check:type-check-coverage, check:type-check-debt --re-measure (on the BUILT workspace — turbo build of all 70 packages first, 70/70 successful), check:query-options-erasure, pnpm lint (rc=0, the blocking ESLint check), and pnpm --filter @objectstack/spec check:generated (13/13 current after gen:docs).\n\nONE RATCHET MOVED, fixed at the source rather than raised: check:query-options-erasure went red — 'packages/objectql/src/engine.ts: erasure count grew 8 -> 9. The file is grandfathered for its EXISTING sites only'. The new site was `this.find(sys_organization, {...} as any)`; the literal type-checks against EngineQueryOptions unmodified, so the cast came out (commit 445359f0b) rather than being laundered through `as unknown as`. Re-run: 'query-options-erasure ratchet holds: 67 unswept non-test site(s) in 17 file(s), none new'. No ceiling touched.\n\nCI status at report time: in_progress — reported at draft-PR time per the 2026-08-10 maintainer decision; the convergence wait is the PM's.",
      "open_questions": [],
      "out_of_scope_findings": [
        "filed as #8928: the binding-point-3 deliverable — an operator-facing inventory of the business identifiers already minted twice, on installs the #8686 backfill cannot repair. Filed unassigned and unlabeled for PM triage, standalone (it is explicitly excluded from this card's completion scope by the ruling, not a sub-task of it). Dedupe search first over the autonumber/tenancy family (#8686, #8844, #5495, #6806, #6249, #8323, #3696, #7287) found no existing card; the nearest thing is backfillSeedTenancy's `collisions` array, which is reachable only during a repair run, only on single-tenant installs, only for the seed split, and is never persisted. The card names the perishability that gives it a deadline: merging the two partitions destroys the organization_id = NULL marker that distinguishes the two rows."
      ]
    }

    Generated by Claude Code


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions