fix(runtime): a self-hosted restart reads the kernel's own sys_metadata back into the registry

.changeset/20071-standalone-hydration.md

@@ -0,0 +1,48 @@
+---
+'@objectstack/runtime': patch
+---
+
+fix(runtime): a self-hosted restart reads the kernel's own `sys_metadata` back into the registry, so objects authored at runtime keep serving
+
+An object created and published at runtime on a self-hosted install (Studio,
+`PUT /api/v1/meta/object/…`, `POST /api/v1/packages/<id>/publish-drafts`)
+answered `404 OBJECT_NOT_FOUND` on the data API after the next restart, while
+its `sys_metadata` row was still there and
+`GET /api/v1/meta/object/<name>/published` still served it. Every `os dev` / `os serve` / `os start` boot was affected.
+
+**Cause.** `createStandaloneStack` stamps `environmentId: 'env_local'` on every
+boot (or whatever `OS_ENVIRONMENT_ID` names), and `ObjectQLPlugin.start()` read
+any environment id as "a per-project kernel whose metadata comes from an
+artifact or a control-plane proxy", so it skipped reading `sys_metadata` at
+boot. It logged `Project kernel — skipping sys_metadata hydration (metadata
+sourced from artifact)`, which was false on this composition. This is the same
+deduction that `runPlatformMigrations` was declared out of.
+
+**Fix: a declaration, not a deduction.** `createStandaloneStack` now declares
+`hydrateMetadataFromDb: true` to `ObjectQLPlugin`, where it used to be deduced
+from the environment-id stamp. The plugin option's own caution holds for every
+boot this function builds, for two reasons:
+
+- the registry is per-instance: the function constructs a fresh
+  `ObjectQLPlugin`, which builds its own `ObjectQL` and `SchemaRegistry`;
+- `sys_metadata` is on the kernel's own driver: it declares no datasource, so it
+  routes to the one `default` datasource the function composes, and every
+  database driver kind the function dispatches is a direct driver, never a
+  control-plane proxy.
+
+What an upgraded install sees at boot:
+
+- every env-wide `sys_metadata` row (`organization_id` NULL), any metadata type,
+  is registered again. Org-scoped rows are still served on demand and are not
+  read at boot (ADR-0005);
+- a stored row that cannot register now says so at boot, on lines that were
+  never reached here before: `[Protocol] [metadata_field_type_refused] …` at
+  `error`, `[Protocol] Failed to hydrate <type>/<name>: …` and
+  `[Protocol] [metadata_spec_invalid] …` at `warn`. The same boot also reports
+  org-scoped rows of types that are not per-org overridable, on one aggregated
+  line. Each line names its remedy;
+- the one-shot `os migrate *` / `os meta *` commands hydrate too, so a plan or
+  a scan covers runtime-authored objects the way the serving boot registers
+  them. The read itself writes nothing, and a deferred-DDL boot
+  (`os migrate plan`, `os migrate duplicates`) still defers the tables of what
+  it read.

packages/cli/test/package-restart-acceptance.integration.test.ts

@@ -29,22 +29,22 @@
  *   2. `GET /api/v1/data/<ns>_<obj>` — the object must be REGISTERED to serve.
  *   3. `GET /api/v1/meta/object/<ns>_<obj>/published` — the stored row.
  *
- * ## What this file finds at the head it landed on
+ * ## What this file pins
  *
- * Probes 1 and 3 cross the restart: the package comes back from `sys_packages`
- * (the `package-registry` mount) and the published row is still served. Probe
- * 2 does NOT: after the restart the data route answers `404 OBJECT_NOT_FOUND`
- * for an object that was published and serving before it. The stock
- * composition is `createStandaloneStack` (`packages/runtime`), which builds
- * `ObjectQLPlugin({ environmentId: 'env_local', … })` without
- * `hydrateMetadataFromDb`, so `ObjectQLPlugin.start()` logs `Project kernel —
- * skipping sys_metadata hydration` and never re-registers a runtime-authored
- * object. The agreement is therefore pinned as `it.fails` (the precedent is
- * `commands.test.ts`): it goes red on the change that makes the three probes
- * agree, and that change promotes it to a plain `it`. Every other assertion
- * below is harness health for it — `it.fails` is green on ANY failure, so the
- * chain, the pre-restart agreement and the two probes that do survive are
- * asserted on their own.
+ * All three probes cross the restart. Probe 1: the package comes back from
+ * `sys_packages` (the `package-registry` mount). Probe 3: the published row is
+ * still served. Probe 2: the data route serves the object again, because the
+ * stock composition, `createStandaloneStack` (`packages/runtime`), DECLARES
+ * `hydrateMetadataFromDb` on its `ObjectQLPlugin` (#20071), so
+ * `ObjectQLPlugin.start()` reads the runtime-authored object back from
+ * `sys_metadata`. Before that declaration the stack stamped `environmentId:
+ * 'env_local'`, the plugin read the stamp as "a per-project kernel" and skipped
+ * the read, and the data route answered `404 OBJECT_NOT_FOUND` after the
+ * restart for an object that was published and serving before it. This file
+ * landed with probe 2 as `it.fails` and went red on that fix, which promoted it
+ * to a plain `it`. The other assertions below are harness health for probe 2:
+ * the chain, the pre-restart agreement and the other two probes are asserted on
+ * their own.
  *
  * Tier: the name carries no nightly tier (`scripts/nightly-tiers.mjs`), so it
  * runs in the per-PR and merge-queue Test Core run; it spawns the CLI, so
@@ -377,20 +377,13 @@ describe('#17676 ruling A\' item 5: the three probes across a restart of a stock
     expect((after?.published.body as { name?: unknown } | null)?.name).toBe(OBJECT);
   });
 
-  // Known-broken at the head this landed on: the data route answers
-  // `404 OBJECT_NOT_FOUND` after the restart, because the stock standalone
-  // composition skips `sys_metadata` hydration (see the header). Promote to a
-  // plain `it` in the change that makes the composition re-register
-  // runtime-authored objects — that is the change that meets #17676's
-  // acceptance, and it is the one that turns this red.
-  it.fails(
-    'probe 2 after the restart — the data route serves the published object, so the three probes agree '
-      + '(known-broken: the stock composition skips sys_metadata hydration; promote to a plain assertion once fixed)',
-    () => {
-      expect(
-        after?.data.status,
-        `GET /data/${OBJECT} after a restart: ${JSON.stringify(after?.data.body)} (code ${String(errorCode(after?.data.body))})`,
-      ).toBe(200);
-    },
-  );
+  // #17676's acceptance. Landed as `it.fails` while the stock standalone
+  // composition skipped `sys_metadata` hydration; promoted on #20071, the change
+  // that declares it (see the header).
+  it('probe 2 after the restart — the data route serves the published object, so the three probes agree', () => {
+    expect(
+      after?.data.status,
+      `GET /data/${OBJECT} after a restart: ${JSON.stringify(after?.data.body)} (code ${String(errorCode(after?.data.body))})`,
+    ).toBe(200);
+  });
 });

packages/runtime/src/standalone-stack-hydrate-metadata.test.ts

@@ -0,0 +1,186 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// [#20071] A standalone boot reads its OWN `sys_metadata` back into the
+// registry, whatever environment id it stamps.
+//
+// `createStandaloneStack` stamps `environmentId: 'env_local'` (or whatever
+// `OS_ENVIRONMENT_ID` / `cfg.environmentId` names), and `ObjectQLPlugin.start()`
+// used to read ANY environment id as "a per-project kernel whose metadata comes
+// from an artifact or a control-plane proxy" and skip Phase-2 hydration. So an
+// object created and published at runtime kept its `sys_metadata` row across a
+// restart and was never registered again: the data API answered
+// `404 OBJECT_NOT_FOUND` for it (measured end to end by
+// `packages/cli/test/package-restart-acceptance.integration.test.ts`).
+//
+// Two halves, pinned where each is observable:
+//
+//   1. The DECLARATION, read off the plugin the stack hands the kernel — the
+//      same seam `standalone-stack-default-environment-id.test.ts` reads the
+//      stamped id from. It is asserted under all three ways an id gets stamped,
+//      because the defect was precisely that the stamp decided hydration.
+//   2. The EFFECT, on a real kernel booted twice over one SQLite file: an
+//      env-wide `object` row AND an env-wide `app` row written through the
+//      protocol on boot 1 are in the second boot's registry, and a record
+//      inserted on boot 1 is read back through the engine on boot 2. The `app`
+//      row is there because the card measured one object only: hydration is
+//      type-blind for env-wide rows (`loadMetaFromDb` registers objects through
+//      `registerObject` and every other type through `registerItem`), and this
+//      makes that reach a measurement instead of a reading of the code.
+//
+// ⛔ Org-scoped rows are NOT asserted here, deliberately: `loadMetaFromDb`
+// hydrates `organization_id IS NULL` rows only (ADR-0005 — a per-org overlay
+// is served on demand, never grafted into the registry every org shares), so an
+// org row absent after a restart is the design, not this defect.
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { ObjectQL } from '@objectstack/objectql';
+import { Runtime } from './runtime.js';
+import { createStandaloneStack } from './standalone-stack.js';
+
+// [#10126] Pay the first transform of these dist-resolved workspace deps at MODULE
+// LOAD. Each is reached below through a dynamic `import()` inside an `it()` body
+// (the factory and the kernel boot both import lazily) -- vitest clocks those,
+// while collection is clocked against nothing. See
+// `scripts/check-test-source-alias.mjs` (the clocked-window rule).
+import '@objectstack/metadata';
+import '@objectstack/objectql';
+import '@objectstack/service-datasource';
+
+const OBJECTQL_PLUGIN = 'com.objectstack.engine.objectql';
+const BOOT_TIMEOUT = 120_000;
+
+const OBJECT = 'hydr_widget';
+const APP = 'hydr_console';
+
+/**
+ * The hydration flag as `ObjectQLPlugin` actually holds it. TypeScript-private,
+ * read on purpose: it is the one value `start()` gates Phase 2 on, so a pin that
+ * re-declared the literal would stay green through a change that stopped
+ * passing it.
+ */
+function hydrationFlag(plugins: readonly unknown[]): { environmentId: unknown; hydrateMetadataFromDb: unknown } {
+    const objectql = plugins.find((p: any) => p?.name === OBJECTQL_PLUGIN) as any;
+    expect(objectql, `stack must carry ${OBJECTQL_PLUGIN}`).toBeDefined();
+    return { environmentId: objectql.environmentId, hydrateMetadataFromDb: objectql.hydrateMetadataFromDb };
+}
+
+async function boot(plugins: readonly unknown[]) {
+    const runtime = new Runtime({ cluster: false });
+    const kernel = runtime.getKernel();
+    for (const p of plugins) await kernel.use(p as any);
+    await kernel.bootstrap();
+    return kernel;
+}
+
+describe('[#20071] createStandaloneStack declares sys_metadata hydration instead of deducing it from the environment id', () => {
+    const dirs: string[] = [];
+    const kernels: any[] = [];
+    let savedEnvId: string | undefined;
+    let savedHome: string | undefined;
+
+    function tempDir(tag: string): string {
+        const dir = mkdtempSync(join(tmpdir(), `os-20071-${tag}-`));
+        dirs.push(dir);
+        return dir;
+    }
+
+    beforeEach(() => {
+        savedEnvId = process.env.OS_ENVIRONMENT_ID;
+        savedHome = process.env.OS_HOME;
+        delete process.env.OS_ENVIRONMENT_ID;
+        process.env.OS_HOME = tempDir('home');
+    });
+
+    afterEach(async () => {
+        for (const k of kernels.splice(0)) {
+            try { await k.shutdown(); } catch { /* noop */ }
+        }
+        if (savedEnvId === undefined) delete process.env.OS_ENVIRONMENT_ID;
+        else process.env.OS_ENVIRONMENT_ID = savedEnvId;
+        if (savedHome === undefined) delete process.env.OS_HOME;
+        else process.env.OS_HOME = savedHome;
+        for (const d of dirs.splice(0)) {
+            try { rmSync(d, { recursive: true, force: true }); } catch { /* noop */ }
+        }
+    });
+
+    it('hands ObjectQLPlugin `hydrateMetadataFromDb: true` under every way an environment id is stamped', async () => {
+        const byDefault = await createStandaloneStack({ databaseUrl: 'memory://issue-20071-default' });
+        expect(hydrationFlag(byDefault.plugins)).toEqual({ environmentId: 'env_local', hydrateMetadataFromDb: true });
+
+        process.env.OS_ENVIRONMENT_ID = 'env_from_the_environment';
+        const byEnv = await createStandaloneStack({ databaseUrl: 'memory://issue-20071-env' });
+        expect(hydrationFlag(byEnv.plugins)).toEqual({
+            environmentId: 'env_from_the_environment',
+            hydrateMetadataFromDb: true,
+        });
+
+        const byConfig = await createStandaloneStack({
+            environmentId: 'env_from_the_config',
+            databaseUrl: 'memory://issue-20071-cfg',
+        });
+        expect(hydrationFlag(byConfig.plugins)).toEqual({
+            environmentId: 'env_from_the_config',
+            hydrateMetadataFromDb: true,
+        });
+    }, BOOT_TIMEOUT);
+
+    it('an env-wide object and an env-wide app written at runtime are registered again after a restart on the same database file', async () => {
+        const dir = tempDir('restart');
+        const databaseUrl = `file:${join(dir, 'hydrate.db')}`;
+        const stackConfig = { projectRoot: dir, databaseUrl, skipSeedData: true, runPlatformMigrations: false } as const;
+
+        // ── boot 1: author both rows at runtime, and one record ────────────────
+        const first = await boot((await createStandaloneStack(stackConfig)).plugins);
+        kernels.push(first);
+        const protocol: any = first.getService('protocol');
+        await protocol.saveMetaItem({
+            type: 'object',
+            name: OBJECT,
+            item: {
+                name: OBJECT,
+                label: 'Widget',
+                sharingModel: 'private',
+                fields: { title: { type: 'text', label: 'Title' } },
+            },
+        });
+        await protocol.saveMetaItem({ type: 'app', name: APP, item: { name: APP, label: 'Hydration Console' } });
+        const engine1 = first.getService<ObjectQL>('objectql');
+        // Harness health: both rows are persisted env-wide and active — exactly
+        // the population `loadMetaFromDb` selects — so their absence from the
+        // next boot's registry can only be the boot's doing.
+        for (const [type, name] of [['object', OBJECT], ['app', APP]] as const) {
+            const stored: Array<{ state?: unknown; organization_id?: unknown }> =
+                await engine1.find('sys_metadata', { where: { type, name } });
+            expect(
+                stored.map((r) => ({ state: r.state, organization_id: r.organization_id ?? null })),
+                `boot 1 persists ${type}/${name} as one env-wide active row`,
+            ).toEqual([{ state: 'active', organization_id: null }]);
+        }
+        // The object is also serving on boot 1 (its write-through is not gated on
+        // the environment id). The app is NOT in this boot's registry: the
+        // protocol's write-through for every non-object type returns early on a
+        // kernel with an environment id, so on this composition boot hydration
+        // is the only path that puts it there — which is what makes it a sharp
+        // second leg below.
+        expect(engine1.registry.getObject(OBJECT)?.name, 'boot 1 registers the object it just saved').toBe(OBJECT);
+        await engine1.insert(OBJECT, { title: 'survives the restart' });
+        await first.shutdown();
+        kernels.splice(kernels.indexOf(first), 1);
+
+        // ── boot 2: same file, nothing authored — only hydration can bring them back
+        const second = await boot((await createStandaloneStack(stackConfig)).plugins);
+        kernels.push(second);
+        const engine2 = second.getService<ObjectQL>('objectql');
+        expect(engine2.registry.getObject(OBJECT)?.name, `${OBJECT} is registered after the restart`).toBe(OBJECT);
+        expect(
+            engine2.registry.getItem<{ name?: string }>('app', APP)?.name,
+            `app ${APP} is registered after the restart`,
+        ).toBe(APP);
+        const rows: Array<{ title?: unknown }> = await engine2.find(OBJECT, {});
+        expect(rows.map((r) => r.title)).toEqual(['survives the restart']);
+    }, BOOT_TIMEOUT);
+});

packages/runtime/src/standalone-stack.ts

@@ -223,8 +223,8 @@ export const StandaloneStackConfigSchema = z.object({
      * Defaults to `true`, and that default is the fix: a standalone kernel
      * OWNS its local platform tables, which is what the gate in
      * `assembleMetadataProtocol` always meant to say. It used to deduce that
-     * from `environmentId === undefined`, and line ~567 below stamps
-     * `'env_local'` on every boot — so the block never ran and #8686's
+     * from `environmentId === undefined`, and `createStandaloneStack` below
+     * stamps `'env_local'` on every boot — so the block never ran and #8686's
      * "covers every existing deployment" half covered no self-hosted install
      * at all.
      *
@@ -754,7 +754,42 @@ export async function createStandaloneStack(config?: StandaloneStackConfig): Pro
         // every self-hosted install. A standalone kernel owns its local
         // platform tables — say so — and let a read-only one-shot boot turn
         // it off explicitly.
-        new ObjectQLPlugin({ environmentId, runPlatformMigrations: cfg.runPlatformMigrations ?? true }),
+        //
+        // [#20071] `hydrateMetadataFromDb` is declared here for the same
+        // reason. `ObjectQLPlugin.start()` reads `sys_metadata` back into the
+        // registry only when `environmentId === undefined` or this option is
+        // set, and read the `'env_local'` stamp above as "a per-project kernel
+        // whose metadata comes from an artifact or a control-plane proxy". So
+        // every self-hosted restart dropped the objects authored at runtime
+        // (Studio, `PUT /api/v1/meta/*`, `publish-drafts`) from the registry
+        // while their rows stayed in the database, and the data API answered
+        // `404 OBJECT_NOT_FOUND` for them.
+        //
+        // The option's own caution — set it ONLY when the kernel's registry is
+        // per-instance isolated AND `sys_metadata` lives on the kernel's own
+        // local driver — holds here, clause by clause:
+        //
+        //   - per-instance registry: this function constructs a fresh
+        //     `ObjectQLPlugin` with no shared `ql`, so its `init()` builds a new
+        //     `ObjectQL`, and each `ObjectQL` owns its `SchemaRegistry`;
+        //   - the kernel's own driver: `sys_metadata` declares no datasource,
+        //     so it routes to the one `default` datasource this function
+        //     composes (`defaultDatasourcePlugin` above), and every
+        //     `databaseDriver` kind dispatched above is a direct driver, never
+        //     a control-plane proxy.
+        //
+        // Both facts are properties of THIS function, not of its caller, so no
+        // caller can make either clause false — which is why it is a literal
+        // and not a config field. The one-shot `os migrate *` / `os meta *`
+        // funnel (`bootSchemaStack`) wants it too: those commands diff and scan
+        // the object set the serving boot registers, and the hydration read
+        // writes nothing (a boot that defers DDL still defers the tables of
+        // what it hydrated).
+        new ObjectQLPlugin({
+            environmentId,
+            runPlatformMigrations: cfg.runPlatformMigrations ?? true,
+            hydrateMetadataFromDb: true,
+        }),
     ];
     if (artifactBundle) {
         plugins.push(new AppPlugin(artifactBundle, undefined, {