Repository navigation
Epic: External Datasource Runtime — implement ADR-0062 (auto-connect, credentials, analytics, cleanup) #2163
Copy link
Copy link
Closed
Description
Tracking/epic issue to drive ADR-0062 — External Datasource Runtime to completion.
- ADR:
docs/adr/0062-external-datasource-runtime.md(merged in docs(adr): ADR-0062 — External Datasource Runtime (connection, credentials, visibility, query completeness) #2162) - Builds on ADR-0015 (federation). Goal: declare an external datasource → it's visible, auto-connected, validated, and queryable with zero app boilerplate, through one connection mechanism shared by code- and runtime-origin datasources, backward-compatible (existing managed apps unchanged).
Phase 0 — shipped ✅
- Read path honors
remoteName/remoteSchema— fix(driver-sql,objectql): honor external.remoteName/remoteSchema in the federation read path (ADR-0015) #2138 - Read path honors
external.columnMap(remote col ≠ local field) — feat(driver-sql): honor external.columnMap on federated objects (ADR-0015) #2149 - Canonical showcase example +
objectstack verifyskip for read-only external — feat(app-showcase): add an end-to-end external datasource + federated object example (#2111) #2139 - Code-defined datasources visible in
/api/v1/datasources+/meta/datasource+ Setup — fix(runtime): surface code-defined datasources on the host-config boot path (ADR-0015 §18) #2157
Phase 1 — DatasourceConnectionService + auto-connect (D1, D2, D5) — core
- D1 New
DatasourceConnectionService: given a datasource def, build a driver via the injected factory, connect, andengine.registerDriverunder the datasource name. One "definition → live driver" path (extract fromstandalone-stack.ts, call fromapp-plugin.ts). - D2 Opt-in-safe gating: connect only when the datasource is
external, or an object/datasourceMappingroutes to it, orautoConnect: true. A managed+unrouted datasource stays metadata-only.examples/app-crmand other existing apps must be byte-for-byte unchanged. - D5 Lifecycle: connect/disconnect (graceful shutdown), per-datasource pool, optional health probe in the admin list; connect before the
kernel:readyvalidation gate; fail-fast forexternal+onMismatch:'fail', degrade-with-warning otherwise. - Refactor the single
default-driver bootstrap onto the same service last (highest-risk; behind tests). - Acceptance: a stack that only declares an external datasource (no
onEnable) can query its federated objects via REST; full dogfood gate green (every example app boots); app-crm behavior unchanged.
Phase 2 — credentials at connect (D3)
- Resolve
external.credentialsRef/secretconfig viaSecretBinderoverICryptoProviderbefore building the driver; converge with the runtime-admin "Add Datasource" secret path. - Fail-closed (clear boot error, datasource left unconnected) when the host can't decrypt; open-core default
InMemoryCryptoProvider. - Acceptance: a Postgres external datasource with
credentialsRefconnects using the decrypted secret; missing/undecryptable secret fails closed with a clear message (no silent skip).
Phase 3 — native-analytics SQL over external objects (D6)
- Route the analytics native-SQL strategy's
FROM/column references through the driver's physical-table/column resolution (remoteName/remoteSchema/columnMap) — reuse, don't re-implement. - Until done, analytics over external objects stays disabled rather than querying the wrong table.
- Acceptance: a dashboard/report over a federated object aggregates against the correct remote table/columns.
Phase 4 — reconciliation + cleanup (D7, D8)
- D7 For external objects,
columnMapis authoritative; rejectfield.columnNameon external objects at validation (no silent dual-source) until a unified column-resolution model exists. Managedfield.columnNameuntouched. - D8 Drop the
onEnabledriver-registration bridge fromexamples/app-showcase(fixture provisioning may remain); keeponEnable+ctx.drivers.registerdocumented as the escape hatch. - Acceptance: showcase declares its external datasource with no
onEnabledriver wiring and still works end-to-end.
Notes
- Each phase = its own PR + changeset (published packages are in the fixed group).
- Phase 1's
default-driver refactor is the riskiest single step (hot boot path for all apps) — land declared-datasource auto-connect first, refactordefaultlast, behind the dogfood gate. - Coordinate with the already-shipped visibility work (fix(runtime): surface code-defined datasources on the host-config boot path (ADR-0015 §18) #2157) — both are "declared datasource as a runtime first-class citizen".
References
- ADR-0062 (docs(adr): ADR-0062 — External Datasource Runtime (connection, credentials, visibility, query completeness) #2162), ADR-0015 (+ §18 addendum)
- Sites:
packages/runtime/src/{standalone-stack,app-plugin}.ts,packages/objectql/src/engine.ts,packages/services/service-datasource/*,packages/plugins/driver-sql/src/sql-driver.ts
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
xuyushun441-sys commentedon Jun 22, 2026
Phase 1 note — leave an open, de-plan-ized connect-policy seam (don't fork for cloud)
Recording a design constraint surfaced while reviewing the open/closed boundary for this epic.
Conclusion: everything in this epic (D1–D8, all phases) is open-core runtime mechanism — it all lands in framework, nothing is EE/closed. ADR-0062's Non-goals already scope out "multi-tenant credential vaulting beyond the SecretBinder seam", so the closed pieces (KMS
SecretBinderimpl, AI introspection, multi-tenant egress/quota governance) live in the cloud repo on top of the seams this epic exposes — not in here.But the multi-tenant host (shared-container cloud) needs to be able to disable external-datasource connect (SSRF / egress isolation) while self-hosted single-env runs it on by default. So Phase 1's
DatasourceConnectionServicemust consult an injectable connect policy before building/registering a driver — not hard-code "always connect".Concretely, when implementing D1/D5:
DatasourceConnectPolicy(or fold it into the D5 connect-error/lifecycle hook) consulted before connect, alongside the existing D2 gate (external/ routed /autoConnect:true).sys_environment.plan+ egress allow-list + per-tenant quota and fail-closes on the shared runtime — that impl stays in the cloud repo.Net: framework gains one open policy interface + a permissive default. Same single connect path for code- and runtime-origin (D1) — the host injects a stricter policy, we do not fork a second connect path for cloud. No cloud-side ADR needed; cloud's policy reuses the existing plan-gating mechanism (the same place that gates the AI authoring assistant today).
feat(datasource): DatasourceConnectionService + declared auto-connect…