fix(service-datasource): in development, a SQLite file that cannot be opened or created fails its connect instead of stepping down to an in-memory engine

.changeset/24950-sqlite-open-failure-no-stepdown.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/service-datasource': patch
+---
+
+fix(service-datasource): in development, a SQLite datasource whose file cannot be opened or created fails its connect, as in production, instead of stepping down to an in-memory engine
+
+Clause-②: no
+
+`resolveSqliteDriver` probes a native `better-sqlite3` connection in development and steps down to wasm SQLite, then to an in-memory engine, when the probe fails. It used to step down on any probe failure. So a declared SQLite datasource whose file could not be opened read `connected` on a development boot (`NODE_ENV=development`, which the source-tree `bin/run-dev.js` sets). Examples: a path under a regular file, a parent directory the process cannot make, a permission refusal, a directory where the file should be. Every query on it then went to an empty in-memory store, so a maintenance command could report a store it never opened as examined and clean.
+
+- **Now:** a probe failure that is the database file failing to open or be created is re-thrown in development, as in production. The datasource's connect fails with the driver's `SqliteDatabaseFileError`, which names the file and the operating system's answer (`EEXIST`, `ENOTDIR`, `ENOENT`, `EACCES`, … or `SQLITE_CANTOPEN`). `listConnectionStates()` reports it as `failed`. This holds for every SQLite datasource, the primary included, so a development boot whose primary database file cannot be opened now stops and names the file.
+- **Unchanged:** a native addon that cannot load (an ABI mismatch, an addon that was never built, a missing `better-sqlite3`) still steps down to wasm SQLite, with the same warning. An absent file in a directory that exists, or that the driver can make, still opens natively and is created. Production behaviour is unchanged.
+
+What to do: if a development boot now refuses a SQLite datasource, fix its `filename`. The boot names the path it could not open.

packages/services/service-datasource/src/__tests__/sqlite-driver-fallback-file-open-failure.test.ts

@@ -0,0 +1,230 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * In development, a SQLite database FILE that cannot be opened or created fails
+ * the probe, as it does in production. Only a native addon that cannot LOAD
+ * steps down to wasm SQLite (#24950).
+ *
+ * Before: `resolveSqliteDriver` stepped down on ANY probe failure in
+ * development. A file under a regular file failed native, failed wasm on the
+ * same path, and was answered by the in-memory rung. So a declared datasource
+ * that could not answer one query read `connected`, and
+ * `os migrate audit-mail-bodies` reported a store it never opened as examined
+ * and clean.
+ *
+ * Real `better-sqlite3`, real wasm SQLite and real files in a scratch
+ * directory, as opposed to `../sqlite-driver-fallback.test.ts`, which mocks
+ * all three driver packages. The one thing faked is an addon that cannot load.
+ * It is faked at the seam where `better-sqlite3` loads its addon: the knex
+ * better-sqlite3 client's `acquireRawConnection` (`new Database(file)`).
+ * The fake throws a REAL load error: Node's own `ERR_DLOPEN_FAILED`, or the
+ * `bindings` package's own "not built" error.
+ */
+
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { createRequire } from 'node:module';
+import { tmpdir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { SqlDriver } from '@objectstack/driver-sql';
+import {
+  NATIVE_SQLITE_WASM_FALLBACK_WARNING,
+  resolveSqliteDriver,
+  type ResolvedSqliteDriver,
+} from '../sqlite-driver-fallback.js';
+import { DatasourceConnectionService } from '../datasource-connection-service.js';
+import { createDefaultDatasourceDriverFactory } from '../default-datasource-driver-factory.js';
+
+let dir: string;
+const opened: ResolvedSqliteDriver[] = [];
+
+beforeEach(() => {
+  dir = mkdtempSync(join(tmpdir(), 'os-24950-'));
+  writeFileSync(join(dir, 'plain'), '');
+});
+
+afterEach(async () => {
+  vi.restoreAllMocks();
+  await Promise.all(opened.splice(0).map((r) => r.driver?.disconnect?.().catch(() => {})));
+  rmSync(dir, { recursive: true, force: true });
+});
+
+async function resolveDev(filename: string, warn = vi.fn()): Promise<ResolvedSqliteDriver> {
+  const resolved = await resolveSqliteDriver({ filename, dev: true, warn });
+  opened.push(resolved);
+  return resolved;
+}
+
+/** `promise`'s rejection; a failure naming the engine when it resolves instead. */
+async function rejectionOf(promise: Promise<ResolvedSqliteDriver>): Promise<any> {
+  let resolved: ResolvedSqliteDriver;
+  try {
+    resolved = await promise;
+  } catch (error) {
+    return error;
+  }
+  opened.push(resolved);
+  throw new Error(`resolved to engine '${resolved.engine}' for a file that cannot be opened or created`);
+}
+
+/** The driver's refusal of the file: its own error name, the system's code, and the file. */
+function expectFileRefusal(error: any, filename: string, code: string): void {
+  expect(error?.name).toBe('SqliteDatabaseFileError');
+  expect(error?.code).toBe(code);
+  expect(error?.filename).toBe(filename);
+}
+
+/** A real `ERR_DLOPEN_FAILED`: Node asked to load, as an addon, a file that is not one. */
+function realDlopenFailure(): Error {
+  const notAnAddon = join(dir, 'not-an-addon.node');
+  writeFileSync(notAnAddon, 'x');
+  try {
+    process.dlopen({ exports: {} }, notAnAddon);
+  } catch (error) {
+    return error as Error;
+  }
+  throw new Error('process.dlopen loaded a file that is not an addon');
+}
+
+/**
+ * The real error `better-sqlite3` throws when its addon was never built: the
+ * `bindings` package it loads the addon through, asked for an addon that is
+ * not there. It carries no code.
+ */
+function realNotBuiltFailure(): Error {
+  const fromHere = createRequire(import.meta.url);
+  const fromDriver = createRequire(fromHere.resolve('@objectstack/driver-sql'));
+  const betterSqlite3 = fromDriver.resolve('better-sqlite3/package.json');
+  const bindings = createRequire(betterSqlite3)('bindings');
+  try {
+    bindings({ bindings: 'os_24950_never_built.node', module_root: dirname(betterSqlite3) });
+  } catch (error) {
+    return error as Error;
+  }
+  throw new Error('bindings located an addon that was never built');
+}
+
+/**
+ * Make the native addon fail to load with `error`, where `better-sqlite3`
+ * loads it: the knex better-sqlite3 client's `acquireRawConnection`, which runs
+ * `new Database(file)`. The method is the client class's own, and wasm SQLite's
+ * client is another class, so the wasm rung is untouched.
+ */
+async function failNativeAddonLoad(error: Error): Promise<void> {
+  const probe = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as any);
+  const clientProto = Object.getPrototypeOf((probe as any).knex.client);
+  await probe.disconnect();
+  expect(Object.prototype.hasOwnProperty.call(clientProto, 'acquireRawConnection')).toBe(true);
+  vi.spyOn(clientProto, 'acquireRawConnection').mockRejectedValue(error);
+}
+
+describe('resolveSqliteDriver in development: a file that cannot be opened or created is not stepped down (#24950)', () => {
+  it('a path under a regular file rejects with the driver\'s refusal, never wasm or memory', async () => {
+    const warn = vi.fn();
+    const at = join(dir, 'plain', 'archive.db');
+    expectFileRefusal(await rejectionOf(resolveDev(at, warn)), at, 'EEXIST');
+
+    const through = join(dir, 'plain', 'sub', 'archive.db');
+    expectFileRefusal(await rejectionOf(resolveDev(through, warn)), through, 'ENOTDIR');
+
+    expect(warn).not.toHaveBeenCalled();
+  });
+
+  it('a directory where the file should be rejects with SQLITE_CANTOPEN', async () => {
+    const filename = join(dir, 'is-a-directory.db');
+    mkdirSync(filename);
+    const warn = vi.fn();
+    expectFileRefusal(await rejectionOf(resolveDev(filename, warn)), filename, 'SQLITE_CANTOPEN');
+    expect(warn).not.toHaveBeenCalled();
+  });
+
+  it('an absent file in a directory that exists still opens native, and is created', async () => {
+    const filename = join(dir, 'fresh.db');
+    const warn = vi.fn();
+    const resolved = await resolveDev(filename, warn);
+    expect(resolved.engine).toBe('better-sqlite3');
+    expect(await resolved.driver.execute('SELECT 1 AS one')).toEqual([{ one: 1 }]);
+    expect(existsSync(filename)).toBe(true);
+    expect(warn).not.toHaveBeenCalled();
+  });
+
+  it('an absent file whose parent directories are missing still opens native: connect() makes them', async () => {
+    const filename = join(dir, 'new', 'deeper', 'fresh.db');
+    const resolved = await resolveDev(filename);
+    expect(resolved.engine).toBe('better-sqlite3');
+    expect(existsSync(filename)).toBe(true);
+  });
+});
+
+describe('resolveSqliteDriver in development: a native addon that cannot load still steps down to wasm (#2229)', () => {
+  it('an ABI mismatch (ERR_DLOPEN_FAILED) steps down to wasm SQLite, with the warning', async () => {
+    const loadError = realDlopenFailure();
+    expect((loadError as any).code).toBe('ERR_DLOPEN_FAILED');
+    await failNativeAddonLoad(loadError);
+
+    const warn = vi.fn();
+    const filename = join(dir, 'dev.db');
+    const resolved = await resolveDev(filename, warn);
+    expect(resolved.engine).toBe('sqlite-wasm');
+    expect(warn).toHaveBeenCalledWith(NATIVE_SQLITE_WASM_FALLBACK_WARNING);
+    expect(await resolved.driver.execute('SELECT 1 AS one')).toEqual([{ one: 1 }]);
+  });
+
+  it('an addon that was never built (no code) steps down to wasm SQLite', async () => {
+    const loadError = realNotBuiltFailure();
+    expect((loadError as any).code).toBeUndefined();
+    await failNativeAddonLoad(loadError);
+
+    const resolved = await resolveDev(join(dir, 'dev.db'));
+    expect(resolved.engine).toBe('sqlite-wasm');
+  });
+
+  it('an addon that cannot load AND a path under a regular file still rejects, never memory', async () => {
+    // connect() makes the file's directory before the first statement loads
+    // the addon, so the native rung refuses the path before the load fails.
+    await failNativeAddonLoad(realDlopenFailure());
+
+    const warn = vi.fn();
+    const at = join(dir, 'plain', 'archive.db');
+    expectFileRefusal(await rejectionOf(resolveDev(at, warn)), at, 'EEXIST');
+    expect(warn).not.toHaveBeenCalled();
+  });
+});
+
+describe('resolveSqliteDriver in production: unchanged', () => {
+  it('a path under a regular file returns native unprobed; its connect() refuses the file', async () => {
+    const warn = vi.fn();
+    const at = join(dir, 'plain', 'archive.db');
+    const resolved = await resolveSqliteDriver({ filename: at, dev: false, warn });
+    opened.push(resolved);
+    expect(resolved.engine).toBe('better-sqlite3');
+    expect(warn).not.toHaveBeenCalled();
+    let refusal: unknown;
+    await resolved.driver.connect().catch((e: unknown) => { refusal = e; });
+    expectFileRefusal(refusal, at, 'EEXIST');
+  });
+});
+
+describe('DatasourceConnectionService in development: a declared SQLite datasource on an unopenable path (#24950)', () => {
+  it('reports a failed connection state, and registers no driver', async () => {
+    const registered: string[] = [];
+    const engine = {
+      registerDriver: (driver: { name?: string }) => { registered.push(String(driver?.name)); },
+      getDriverByName: (name: string) => (registered.includes(name) ? {} : undefined),
+    };
+    const service = new DatasourceConnectionService({
+      factory: () => createDefaultDatasourceDriverFactory({ dev: true }),
+      engine: () => engine as any,
+    });
+    const filename = join(dir, 'plain', 'archive.db');
+
+    await service.connectDeclared({
+      datasources: [{ name: 'archive', driver: 'sqlite', config: { filename }, autoConnect: true }],
+    });
+
+    const state = service.getConnectionState('archive');
+    expect(state?.availability).toBe('failed');
+    expect(state?.reason).toContain(filename);
+    expect(registered).toEqual([]);
+  });
+});

packages/services/service-datasource/src/sqlite-driver-fallback.ts

@@ -31,6 +31,39 @@
  * so it surfaces loudly (fail-closed) instead of an operator unknowingly running
  * on wasm/mingo. This mirrors the existing `serve.ts` default-dev fallback and
  * hoists it into one place shared by every sqlite construction site.
+ *
+ * ## Which probe failures step down, and which do not (#24950)
+ *
+ * The step-down exists for ONE cause: the native addon failing to LOAD. That is
+ * the only failure a different engine can cure. In development, then:
+ *
+ *   - **The addon cannot load — steps down.** An ABI mismatch (Node's
+ *     `ERR_DLOPEN_FAILED`, a `NODE_MODULE_VERSION` message), an addon that was
+ *     never built (`bindings`' "Could not locate the bindings file", which has
+ *     no code), or a missing `better-sqlite3` package. Today's path,
+ *     unchanged.
+ *   - **The database FILE cannot be opened or created — re-thrown, as in
+ *     production.** A path under a regular file, a parent directory the driver
+ *     cannot make, a permission refusal, a directory where the file should be.
+ *     `SqlDriver.connect()` names these with the operating system's code
+ *     (`EEXIST`, `ENOTDIR`, `ENOENT`, `EACCES`, …) or SQLite's
+ *     `SQLITE_CANTOPEN`; see {@link isSqliteFileOpenFailure}. Every rung opens
+ *     the same file, so no rung can cure this. Before, the step-down answered
+ *     in its place: the in-memory rung when wasm failed on the same path, or a
+ *     wasm driver whose connect resolved over a file it could not read. So a
+ *     declared datasource that could not answer one query read `connected`,
+ *     and a maintenance command reported a store it never opened as examined
+ *     and clean. Now the datasource's connect fails and its connection state
+ *     says so. That holds for every datasource, the primary included.
+ *     `connect()` makes the directory before the addon loads, so a path the
+ *     driver cannot make is refused at the native rung even when the addon
+ *     could not have loaded.
+ *
+ * An ABSENT file in a directory that exists, or can be made, is not an open
+ * failure: the driver makes the directory and SQLite creates the file. A dev
+ * boot on a fresh checkout keeps working.
+ *
+ * Every other probe failure keeps today's path.
  */
 
 import type { SqliteAbsentFileMode } from '@objectstack/driver-sql';
@@ -100,9 +133,50 @@ function isEphemeralFilename(filename: string): boolean {
   return filename === ':memory:' || filename.startsWith(':');
 }
 
+/**
+ * Is this probe failure the database FILE failing to open or be created, as
+ * opposed to the native addon failing to load (#24950)? Judged on the thrown
+ * value's own `code` alone, never on its message. `SqlDriver.connect()` is what
+ * refuses such a file, with a `SqliteDatabaseFileError` whose `code` is the
+ * answer it got:
+ *
+ *   - an operating-system error code (`EEXIST`, `ENOTDIR`, `ENOENT`, `EACCES`,
+ *     `EPERM`, `EROFS`, …), from making the file's directory. These codes are
+ *     `E` plus capital letters and digits, with no underscore;
+ *   - SQLite's `SQLITE_CANTOPEN`: the directory is there, and SQLite still
+ *     cannot open the file. Only a LOADED addon can answer with a SQLite code.
+ *
+ * An addon that cannot load never carries either. It surfaces at the probe's
+ * `SELECT 1`, whose failure `SqlDriver.execute()` hands back as
+ * `DATABASE_ERROR`, with the load error as its `cause`: Node's
+ * `ERR_DLOPEN_FAILED`, `MODULE_NOT_FOUND`, or no code at all (`bindings`'
+ * "Could not locate the bindings file"). A missing `better-sqlite3` package
+ * fails at construction with knex's install hint, which has no code. None of
+ * these matches.
+ *
+ * ⛔ No message text is read. A failure this misses keeps today's path.
+ */
+function isSqliteFileOpenFailure(err: unknown): boolean {
+  const code = (err as { code?: unknown } | null | undefined)?.code;
+  if (typeof code !== 'string') return false;
+  return /^E[A-Z0-9]+$/.test(code) || code === 'SQLITE_CANTOPEN';
+}
+
+/** Best-effort close of a probe driver that failed; its own error is not the answer. */
+async function closeQuietly(driver: any): Promise<void> {
+  if (typeof driver?.disconnect !== 'function') return;
+  try {
+    await driver.disconnect();
+  } catch {
+    /* ignore */
+  }
+}
+
 /**
  * Probe a `better-sqlite3` SQLite construction and, in dev, step down to wasm
- * SQLite (then in-memory) when the native addon cannot load.
+ * SQLite (then in-memory) when the native addon cannot load. A database file
+ * that cannot be opened or created is re-thrown instead, in dev as in
+ * production (#24950).
  *
  * @see {@link ResolveSqliteDriverOptions}
  */
@@ -149,29 +223,29 @@ export async function resolveSqliteDriver(
     return { driver: buildNative(), engine: 'better-sqlite3', label: 'SqlDriver(sqlite)' };
   }
 
-  // ── Dev: probe-by-connect, step down on native ABI/load failure. ──────────
+  // ── Dev: probe-by-connect, step down on native ABI/load failure only. ─────
 
   // 1. Native better-sqlite3.
   let nativeDriver: any;
   let nativeOk = false;
   try {
     nativeDriver = buildNative();
-    // connect() runs mkdir (so a SELECT on a file DB whose dir is missing does
-    // not false-positive as an ABI failure) + a PRAGMA whose error it swallows;
-    // the raw SELECT 1 below is what reliably forces the native addon to load
-    // and PROPAGATES an ABI mismatch.
+    // connect() makes the file's directory (so a missing directory is not an
+    // open failure) and refuses a file it cannot open or create, naming it; a
+    // PRAGMA whose load error it swallows follows. The raw SELECT 1 below is
+    // what reliably forces the native addon to load and PROPAGATES an ABI
+    // mismatch.
     await nativeDriver.connect();
     await nativeDriver.execute('SELECT 1');
     nativeOk = true;
-  } catch {
+  } catch (err) {
     nativeOk = false;
-    if (typeof nativeDriver?.disconnect === 'function') {
-      try {
-        await nativeDriver.disconnect();
-      } catch {
-        /* ignore */
-      }
-    }
+    await closeQuietly(nativeDriver);
+    // The FILE cannot be opened or created: no other engine can cure that, so
+    // the datasource's connect fails here, as it does in production (#24950).
+    // Every other failure, a native addon that cannot load first among them,
+    // goes on down the chain as before.
+    if (isSqliteFileOpenFailure(err)) throw err;
   }
   if (nativeOk) {
     return { driver: nativeDriver, engine: 'better-sqlite3', label: 'SqlDriver(sqlite)' };
@@ -193,13 +267,7 @@ export async function resolveSqliteDriver(
     wasmOk = true;
   } catch {
     wasmOk = false;
-    if (typeof wasmDriver?.disconnect === 'function') {
-      try {
-        await wasmDriver.disconnect();
-      } catch {
-        /* ignore */
-      }
-    }
+    await closeQuietly(wasmDriver);
   }
   if (wasmOk) {
     warn(NATIVE_SQLITE_WASM_FALLBACK_WARNING);