Skip to content

Epic: External Datasource Runtime — implement ADR-0062 (auto-connect, credentials, analytics, cleanup) #2163

Closed

Description

@xuyushun441-sys

Tracking/epic issue to drive ADR-0062 — External Datasource Runtime to completion.

Phase 0 — shipped ✅

Phase 1 — DatasourceConnectionService + auto-connect (D1, D2, D5) — core

  • D1 New DatasourceConnectionService: given a datasource def, build a driver via the injected factory, connect, and engine.registerDriver under the datasource name. One "definition → live driver" path (extract from standalone-stack.ts, call from app-plugin.ts).
  • D2 Opt-in-safe gating: connect only when the datasource is external, or an object/datasourceMapping routes to it, or autoConnect: true. A managed+unrouted datasource stays metadata-only. examples/app-crm and other existing apps must be byte-for-byte unchanged.
  • D5 Lifecycle: connect/disconnect (graceful shutdown), per-datasource pool, optional health probe in the admin list; connect before the kernel:ready validation gate; fail-fast for external + onMismatch:'fail', degrade-with-warning otherwise.
  • Refactor the single default-driver bootstrap onto the same service last (highest-risk; behind tests).
  • Acceptance: a stack that only declares an external datasource (no onEnable) can query its federated objects via REST; full dogfood gate green (every example app boots); app-crm behavior unchanged.

Phase 2 — credentials at connect (D3)

  • Resolve external.credentialsRef / secret config via SecretBinder over ICryptoProvider before building the driver; converge with the runtime-admin "Add Datasource" secret path.
  • Fail-closed (clear boot error, datasource left unconnected) when the host can't decrypt; open-core default InMemoryCryptoProvider.
  • Acceptance: a Postgres external datasource with credentialsRef connects using the decrypted secret; missing/undecryptable secret fails closed with a clear message (no silent skip).

Phase 3 — native-analytics SQL over external objects (D6)

  • Route the analytics native-SQL strategy's FROM/column references through the driver's physical-table/column resolution (remoteName/remoteSchema/columnMap) — reuse, don't re-implement.
  • Until done, analytics over external objects stays disabled rather than querying the wrong table.
  • Acceptance: a dashboard/report over a federated object aggregates against the correct remote table/columns.

Phase 4 — reconciliation + cleanup (D7, D8)

  • D7 For external objects, columnMap is authoritative; reject field.columnName on external objects at validation (no silent dual-source) until a unified column-resolution model exists. Managed field.columnName untouched.
  • D8 Drop the onEnable driver-registration bridge from examples/app-showcase (fixture provisioning may remain); keep onEnable + ctx.drivers.register documented as the escape hatch.
  • Acceptance: showcase declares its external datasource with no onEnable driver wiring and still works end-to-end.

Notes

  • Each phase = its own PR + changeset (published packages are in the fixed group).
  • Phase 1's default-driver refactor is the riskiest single step (hot boot path for all apps) — land declared-datasource auto-connect first, refactor default last, behind the dogfood gate.
  • Coordinate with the already-shipped visibility work (fix(runtime): surface code-defined datasources on the host-config boot path (ADR-0015 §18) #2157) — both are "declared datasource as a runtime first-class citizen".

References

Activity

  1. xuyushun441-sys commented on Jun 22, 2026

    @xuyushun441-sys
    CollaboratorAuthor

    Phase 1 note — leave an open, de-plan-ized connect-policy seam (don't fork for cloud)

    Recording a design constraint surfaced while reviewing the open/closed boundary for this epic.

    Conclusion: everything in this epic (D1–D8, all phases) is open-core runtime mechanism — it all lands in framework, nothing is EE/closed. ADR-0062's Non-goals already scope out "multi-tenant credential vaulting beyond the SecretBinder seam", so the closed pieces (KMS SecretBinder impl, AI introspection, multi-tenant egress/quota governance) live in the cloud repo on top of the seams this epic exposes — not in here.

    But the multi-tenant host (shared-container cloud) needs to be able to disable external-datasource connect (SSRF / egress isolation) while self-hosted single-env runs it on by default. So Phase 1's DatasourceConnectionService must consult an injectable connect policy before building/registering a driver — not hard-code "always connect".

    Concretely, when implementing D1/D5:

    • Add an injectable DatasourceConnectPolicy (or fold it into the D5 connect-error/lifecycle hook) consulted before connect, alongside the existing D2 gate (external / routed / autoConnect:true).
    • Open-core default impl = allow (subject to D2) → self-host works out of the box, de-plan-ized, no plan coupling in framework.
    • The host can bind a stricter policy. Cloud will bind one that checks sys_environment.plan + egress allow-list + per-tenant quota and fail-closes on the shared runtime — that impl stays in the cloud repo.

    Net: framework gains one open policy interface + a permissive default. Same single connect path for code- and runtime-origin (D1) — the host injects a stricter policy, we do not fork a second connect path for cloud. No cloud-side ADR needed; cloud's policy reuses the existing plan-gating mechanism (the same place that gates the AI authoring assistant today).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions