Skip to content

QA run · integration-system (16/20) · 5b14b7f3 · 2026-10-19 · 11 PASS / 5 PARTIAL / 0 FAIL / 0 BLOCKED / 4 NOT-RUN #24575

Closed as not planned
Closed as not planned
QA run · integration-system (16/20) · 5b14b7f3 · 2026-10-19 · 11 PASS / 5 PARTIAL / 0 FAIL / 0 BLOCKED / 4 NOT-RUN#24575
Labels
qa-run

Description

@objectstack-fleet
objectstack-fleet
bot
opened on Oct 19, 2026

Run record for the 18.0 release sweep, selector integration-system (resolved with node scripts/checklist-select.mjs area:integration-system --json), plan B (Tier 1 + Tier 2; Tier 3 deferred).

Environment fingerprint

  • framework: 5b14b7f3ad8e2e0786d228772f473d07b36b2477 (objectstack-ai/objectstack main), working tree clean
  • console: .objectui-sha = 4016370af3c9c9dafec2bc5299afd8e8079701b8, built with pnpm objectui:build; packages/console/dist/.objectui-sha stamp equals the pin (pnpm check:console-sha green), so the stale-console-bundle trap is ruled out for every console observation
  • app: showcase (examples/app-showcase), canonical boot OS_PORT=PORT pnpm -C examples/app-showcase exec objectstack dev --ui --seed-admin -p PORT -d file:/tmp/qa18/ITEM/data.db; every hand-driven item had its own port and file DB (listed per item below); seed: --seed-admin (admin@objectos.ai) + showcase seed + per-item scratch fixtures
  • Tier 1 pins: vitest per package against the same tree (pnpm exec vitest run FILES), 97 framework files / 2209 tests, 0 failed; objectui-side pins run inside a clone of objectui at 4016370af3c9 (unit vitest + playwright.live.config.ts against a showcase boot on :47431 through the objectui console dev server), RUNNER option (b)
  • browser: Playwright with /opt/pw-browsers/chromium, own context per persona, sign-in through the form
  • timestamps: 2026-10-19, environment prep 12:26–12:43 UTC, testing 12:43–13:53 UTC; Tier 2 items were driven in parallel by sub-runners (≤ 3–4 browser items at once), each on its own port/DB

Scope

Selector integration-system → 20 runnable item(s). Blocked items hidden by the resolver were not run (no --include-blocked).

Tiering (checklist-test SKILL §1): Tier 1 = item carries automated.ref (pin run, cited); Tier 2 = unpinned P0/P1 (hand-driven); Tier 3 = unpinned P2 (deferred, not-run).

item priority surface tier revision item verdict
integration-system.connector-declarative-boot P1 api 1 2 pass
integration-system.connector-degraded-recovery P1 api 2 2 pass
integration-system.connector-stdio-default-deny P1 cli 1 2 pass
integration-system.connector-spec-path-no-escape P2 build 3 2 not-run
integration-system.connector-descriptor-audit P2 cli 1 2 pass
integration-system.flow-connector-picker P2 browser 3 5 not-run
integration-system.webhook-lifecycle P1 mixed 1 7 pass
integration-system.job-scheduled-run P1 mixed 2 4 pass
integration-system.email-template-render P2 mixed 1 7 pass
integration-system.notify-inbox-delivery P1 mixed 2 4 partial
integration-system.external-datasource-federated-read P1 api 2 3 pass
integration-system.datasource-admin-lifecycle P1 api 1 3 partial
integration-system.external-schema-introspection P1 api 1 3 partial
integration-system.external-schema-drift-gate P1 cli 1 2 partial
integration-system.notification-preference-suppression P2 mixed 3 3 not-run
integration-system.datasource-credential-refusal-matrix P0 mixed 2 4 pass
integration-system.connector-auth-kind-application P1 api 1 1 partial
integration-system.external-schema-browser-ui P1 mixed 1 3 not-run
integration-system.inbound-mailbox-association P2 mixed 1 1 pass
integration-system.notification-org-and-redeliver-eligibility P2 api 1 1 pass

Per-clause verdicts

A = acceptance[n], N = negative[n]. Tier 1 rule: a clause is pass only when a passing pin covers it (server-side oracles, or clauses the ref names); clauses a ref says it does not cover, and UI-oracle clauses a server-side pin cannot see, are not-run. Evidence is text from the oracle — no images.

integration-system.connector-declarative-boot — pass (Tier 1 pin)

clause verdict evidence
A0–N1 pass pin: showcase-declarative-mcp.dogfood.test.ts 2/2 passed

integration-system.connector-degraded-recovery — pass (hand-driven (port 47401 · file:/tmp/qa18/conn-degraded/data.db · scratch copy of examples/app-showcase with scripts/mcp-fixture.mjs moved out (twin leg: transport.kind carrier_pigeon)))

clause verdict evidence
A0 pass log: Connector registered DEGRADED: showcase_mcp_tools … MCP error -32000: Connection closed; GET /api/v1/automation/connectors → showcase_mcp_tools {state:degraded, degradedReason, actions:[]} beside 4 ready
A1 pass broken upstream degrades and serves; config-fault twin aborts boot: failed to materialize connector instance … transport.kind must be stdio or http; process exits
A2 pass POST /api/v1/automation/showcase_mcp_connector_echo/trigger → 400 FLOW_FAILED in 0.054s naming the degraded connector and stored reason; run failed with NODE_FAILURE in 3ms
A3 pass retry intervals +5.06s / +10.05s / +20.25s / +40.09s … +160.6s (attempt 6): doubling from 5s, no hot loop; a metadata reload retried 1.2s after publish (see accuracy finding on the verify wording)
A4 pass fixture restored with no restart (same PID); next retry logs Connector showcase_mcp_tools replaced; descriptor ready; echo trigger 200 {upper:OBJECTSTACK}
N0–N1 pass boot survived the dead upstream; degraded husk stays visible in GET /api/v1/automation/connectors

integration-system.connector-stdio-default-deny — pass (Tier 1 pin)

clause verdict evidence
A0–N1 pass pin: mcp-provider.test.ts 19/19 passed

integration-system.connector-spec-path-no-escape — not-run (Tier 3)

clause verdict evidence
A0–A3, N0–N1 not-run Tier 3 deferred by the 18.0 release-sweep plan

integration-system.connector-descriptor-audit — pass (Tier 1 pin)

clause verdict evidence
A0–N1 pass pin: connector-descriptor-audit.test.ts 9/9 passed

integration-system.flow-connector-picker — not-run (Tier 3)

clause verdict evidence
A0–A2, N0–N0 not-run Tier 3 deferred by the 18.0 release-sweep plan

integration-system.webhook-lifecycle — pass (Tier 1 pin)

clause verdict evidence
A0–N2 pass pin: webhook-materialization.dogfood.test.ts 1/1 passed

integration-system.job-scheduled-run — pass (hand-driven (port 47491 · file:/tmp/qa18/job-scheduled/data.db · OS_AUTOMATION_SCHEDULED_WORK_ENABLED=1 · scratch copy of examples/app-showcase with scratch jobs · wall clock before first request 2026-10-19T13:37:16Z))

clause verdict evidence
A0 pass banner 32 flow(s) 21 bound and GET /api/v1/automation/_status bound:true; interval job (15 s) ran repeatedly (run_count 37); once job ran exactly once at 13:43:00.000Z and not again after two restarts; cron with timezone ('48 21 * * *' Asia/Shanghai) fired at 13:48:00.003Z UTC = 21:48 Shanghai; each with its own side-effect row
A1 pass sys_job rows keyed by name with adapter-minted ids; run_count / failure_count / last_status / last_error / last_run_at move with runs; each execution writes a sys_job_run row (status, timings, attempt, trigger, error)
A2 pass retryPolicy maxRetries 2, backoffMs 1000, multiplier 2 → attempts 1/2/3 at +1.01s and +2.01s, all failed; a job without a policy has exactly one attempt
A3 pass timeoutMs 1000 with a 4 s handler → status timeout, duration_ms 1001, error Job "qa_slow" timed out after 1000ms; with a retry policy the timeout retried (attempt 2 at +1.5s), so timeouts count as failures
A4 pass handler missing → WARN [AppPlugin] job handler not found in bundle.functions — skipping {job, handler}; enabled:false → DEBUG job disabled — skipping; neither has a sys_job / sys_job_run / side-effect row
A5 pass objectstack build exits 2 for id: (job.id removed in 17.0.0, name is the identity), cron: (Did you mean schedule?), fn + a stray key; PUT /api/v1/meta/job/qa_runtime_job → 403 NOT_CREATABLE (code-only); POST /api/v1/meta/job → 405; POST /api/v1/data/sys_job → 405; no row created
N0–N1 pass every declared job ran or was skipped with a log line; no runtime-created job row saved

integration-system.email-template-render — pass (Tier 1 pin)

clause verdict evidence
A0–N2 pass pin: email-template-materialization.dogfood.test.ts 4/4 passed

integration-system.notify-inbox-delivery — partial (hand-driven (port 47492 · file:/tmp/qa18/notify-inbox/data.db · assignee and third member via admin create-user; identities via get-session))

clause verdict evidence
A0 pass assignee GET /api/v1/notifications empty before; admin PATCH /api/v1/data/showcase_task/ID {assignee: assignee user id} → 200; assignee list then holds type task.assigned New task assigned: Warehouse schema, read false, actionUrl to the task; unreadCount 1; backing sys_inbox_message rows present
A1 pass third member's list empty while the assignee's lists 2; forged ?userId= / user_id= → empty; forged POST /api/v1/notifications/read with the assignee's id → readCount 0 (receipt unchanged); third member GET /api/v1/data/sys_inbox_message → total 0
A2 pass assignee POST /api/v1/notifications/read → readCount 1, receipt delivered → read; POST /api/v1/notifications/read/all → readCount 1; ?read=false → [] unreadCount 0
A3 blocked(fixture) 401 half holds (anonymous GET /api/v1/notifications → 401 UNAUTHENTICATED); the 501 half needs a boot without MessagingServicePlugin, which no objectstack dev boot is (the item's knownGaps[2])
A4 pass runtime flow with a push-channel notify → trigger 200, node reports selected 1, acted 0; WARN [messaging] emit: channel 'push' is not registered; refused 1 delivery row(s) …; delivery and inbox counts unchanged before/after
N0–N1 pass flow-driven notify reached the listing; push reported acted 0, not a fake delivery; no cross-user visibility

integration-system.external-datasource-federated-read — pass (hand-driven (port 47402 · file:/tmp/qa18/ext-fed/data.db))

clause verdict evidence
A0 pass showcase_ext_customer 3 rows; showcase_ext_order total 4; filter {status:paid} → 3 vs 4 unfiltered; top=2 → 2 rows, total 4, hasMore true; OData $filter → 400 INVALID_FILTER
A1 pass POST / PATCH / DELETE → 403 EXTERNAL_WRITE_FORBIDDEN (allowWrites false, writable false); re-reads unchanged
A2 pass GET /api/v1/datasources lists showcase_external {schemaMode external, origin code, status ok}; Setup → Datasources shows both rows connected
N0 pass no write appeared to succeed

integration-system.datasource-admin-lifecycle — partial (Tier 1 pin)

clause verdict evidence
A0–A6 not-run not covered by pin (ref: pins route behavior + envelope in-process; the LIVE-mount half is not pinned); not hand-driven in this Tier 1 pass
A7 pass pin: admin-routes.test.ts 30/30 passed; envelope.conformance.test.ts 24/24 passed
A8–N5 not-run not covered by pin (ref: pins route behavior + envelope in-process; the LIVE-mount half is not pinned); not hand-driven in this Tier 1 pass

integration-system.external-schema-introspection — partial (Tier 1 pin)

clause verdict evidence
A0 not-run not covered by pin (ref: #7955 request shape + route twin pinned through HonoHttpServer; the LIVE showcase/sqlite introspection half is not pinned); not hand-driven in this Tier 1 pass
A1 pass pin: remote-tables-twin.equivalence.test.ts 10/10 passed; admin-routes.test.ts 30/30 passed
A2 not-run not covered by pin (ref: #7955 request shape + route twin pinned through HonoHttpServer; the LIVE showcase/sqlite introspection half is not pinned); not hand-driven in this Tier 1 pass
A3 pass pin: remote-tables-twin.equivalence.test.ts 10/10 passed; admin-routes.test.ts 30/30 passed
A4–N3 not-run not covered by pin (ref: #7955 request shape + route twin pinned through HonoHttpServer; the LIVE showcase/sqlite introspection half is not pinned); not hand-driven in this Tier 1 pass

integration-system.external-schema-drift-gate — partial (Tier 1 pin)

clause verdict evidence
A0 not-run not covered by pin (ref: fail/warn/ignore, checkOnBoot gate, drift-event, withheld all-clear pinned; the LIVE boot-abort arm is not pinned); not hand-driven in this Tier 1 pass
A1–A6 pass pin: external-validation-plugin.test.ts 17/17 passed; external-validation-checkonboot.test.ts 13/13 passed; list-diagnosed-consumer-sweep.test.ts 9/9 passed
N0 not-run not covered by pin (ref: fail/warn/ignore, checkOnBoot gate, drift-event, withheld all-clear pinned; the LIVE boot-abort arm is not pinned); not hand-driven in this Tier 1 pass
N1–N5 pass pin: external-validation-plugin.test.ts 17/17 passed; external-validation-checkonboot.test.ts 13/13 passed; list-diagnosed-consumer-sweep.test.ts 9/9 passed

integration-system.notification-preference-suppression — not-run (Tier 3)

clause verdict evidence
A0–A4, N0–N1 not-run Tier 3 deferred by the 18.0 release-sweep plan

integration-system.datasource-credential-refusal-matrix — pass (hand-driven (port 47411 · file:/tmp/qa18/ds-cred/data.db · plus a private Postgres 16 on 127.0.0.1:47412 (scram-sha-256, role with password)))

clause verdict evidence
A0 pass every inline-credential door (config password on pg/mysql/mongo, turso authToken, URL userinfo incl. turso syncUrl, ?password= / ?authToken=, mongo options.auth.password, nested options token/password, dollar-brace placeholders, connector inline authentication, top-level passwd) refused at BOTH doors: os build exit 2 (defineStack validation failed) and POST /api/v1/datasources 400 DATASOURCE_ADMIN_ERROR (also PUT /meta/datasource 422, PUT /meta/connector 422 INVALID_METADATA); messages name external.credentialsRef / auth.credentialRef; turso encryptionKey accepted as the item's knownGap states
A1 pass live credential-protected Postgres: POST /api/v1/datasources {config without password, secret} → 201 status ok; POST /api/v1/datasources/live_pg/test → ok:true tableCount 1; remote-tables lists widget; URL-branch twin ok; no-secret control → status error (SASL); credentialsRef form builds green
A2 pass admin door and /api/v1/meta/datasource door serve identical redacted config; 0 grep hits for planted values across list/detail/data sys_metadata/boot logs/raw db
A3 pass hasSecret true vs false distinguishes set/unset; redactedConfigKeys [encryptionKey] vs []
A4 pass legacy rows planted by direct sys_metadata insert (passwd/pwd, token/jwt/auth_token/authtoken, url userinfo + password, mongo option passwords, ?password=); boot logged metadata_spec_invalid (rows loaded); both doors redact every planted value
A5 pass unknown driver acme_kv: canonical keys, aliases and url userinfo redacted on both doors; apiSecret served as written (ruled boundary #21921)
A6 pass GET then PATCH untouched → 200, stored row byte-identical, probe ok:true; console Edit→Save untouched sends no secret key; carry-forward of encryptionKey/passthrough holds
A7 pass identities proven via GET /api/v1/auth/get-session; auditor.demo and a minted member → 403 PERMISSION_DENIED (manage_platform_settings) on every read and forged write; anonymous → 401; MCP query_records on sys_metadata refused
N0–N6 pass no cleartext on any door; no two-door disagreement; identical redaction on both doors incl. legacy rows; all alias spellings redacted; plugin driver redacted; mask never served; every refusal names credentialsRef

integration-system.connector-auth-kind-application — partial (Tier 1 pin)

clause verdict evidence
A0–A2 pass pin: rest-connector.test.ts 10/10 passed; connector-materialization.test.ts 50/50 passed
A3–N1 not-run not covered by pin (ref: per-kind wire shapes via stubbed fetch + credentialRef resolution and boot-fatal missing ref pinned; LIVE echo-upstream leg, oauth2 authoring refusal and served-body checks not pinned); not hand-driven in this Tier 1 pass
N2 pass pin: rest-connector.test.ts 10/10 passed; connector-materialization.test.ts 50/50 passed

integration-system.external-schema-browser-ui — not-run (Tier 1 pin)

clause verdict evidence
A0–A6 not-run not covered by a passing pin (objectui@4016370a external/api.test.ts 16/16 — pins the API-layer 503 mapping only; acc[6]'s rendered state + server envelope not pinned (partial clause, not scored pass)); not hand-driven in this Tier 1 pass
N0–N4 not-run negative spans the UI half not driven in this Tier 1 pass

integration-system.inbound-mailbox-association — pass (Tier 1 pin)

clause verdict evidence
A0–A3 pass pin: mailbox-service-kernel.test.ts 1/1 passed; mailbox-sync-kernel.test.ts 4/4 passed; inline-forward.test.ts 24/24 passed; mailbox-attachments-kernel.test.ts 7/7 passed; field-mail-association.test.ts 16/16 passed

integration-system.notification-org-and-redeliver-eligibility — pass (Tier 1 pin)

clause verdict evidence
A0–A3 pass pin: notification-organization-follows-event.integration.test.ts 4/4 passed; messaging-service.test.ts 79/79 passed; webhook-redeliver-member.test.ts 18/18 passed; mcp-stdio-cwd.test.ts 7/7 passed

Reproduction rules

No fail in this selector. The P0 integration-system.datasource-credential-refusal-matrix passed on all 8 acceptance clauses and 7 negatives (first run to score acceptance[1] and [6], against a local credential-protected Postgres); no rule-7 re-derivation was owed.

Derived item verdicts

11 pass · 5 partial · 4 not-run. Partials: datasource-admin-lifecycle, external-schema-introspection, external-schema-drift-gate, connector-auth-kind-application — each ref states that its LIVE half is not pinned, and those halves were not hand-driven in this Tier 1 pass — plus notify-inbox-delivery (A3's 501 half blocked by fixture). Not-run: external-schema-browser-ui (its only pin covers half of acceptance[6]: the API-layer 503 mapping at objectui 4016370a, 16/16 green) and three Tier 3 items.

Checklist-accuracy findings / fixture gaps (for the wave anchor card)

  • datasource-credential-refusal-matrix: the knownGap "no Postgres recipe is proven / the QA container has none" is stale — Postgres 16 is installed at /usr/lib/postgresql/16/bin; working recipe: initdb -A scram-sha-256 as postgres, pg_ctl … -o '-p PORT -k DIR -c listen_addresses=127.0.0.1' start, create a login role with a password and its database, bind through POST /api/v1/datasources with secret (parent dirs must be traversable by postgres). Hardening observation (latent, not a fail under the declared verify — the mask is never served by a read door): PATCH /api/v1/datasources/NAME with secret set to the literal mask string is accepted 200 and rewraps the stored secret to the mask (the reconnect then fails), and PATCH with config.encryptionKey set to the mask overwrites the stored key — no echoed-mask guard in service-datasource's update path; acceptance[6]'s "writing the mask value back verbatim" presumes a served mask. The metadata door serves the opaque external.credentialsRef handle while the admin door serves only hasSecret — say whether intended. A third read door, GET /api/v1/data/sys_metadata (admin only), also redacts — name it. turso is absent from GET /api/v1/datasources/drivers though POST accepts it, and the console edit form for turso shows only Label. Steps nits: schemaMode: "external" on POST needs an external block; the console route is /_console/apps/com.objectstack.setup/metadata/datasource.
  • connector-degraded-recovery: acceptance[3].verify and step 7 say a reload reconcile "resets the backoff" — degradeConnectorInstance (service-automation plugin.ts) resets attempts only on a config-signature change; a reload retries immediately but keeps counting (measured: attempt 6, next delay 160 s). Split the wording (reload = immediate retry; config edit = reset). Step 7's "touch/save the entry" restarts os dev; a restart-free reload vehicle is PUT /api/v1/meta/view/NAME?mode=draft then POST /api/v1/meta/view/NAME/publish. Steps 1 and 6 need a scratch app copy (the stdio fixture path is cwd-relative). The info line [Automation] recovered connector instance … is absent at the default log level.
  • external-datasource-federated-read: acceptance[2].oracle still reads screenshot though rev 2 made the API the oracle; the Setup route is /_console/apps/setup/metadata/datasource (/_console/apps/setup/datasources → "Object Not Found"); the external fixture DB lives in the shared tree (examples/app-showcase/.objectstack/data/showcase_external.db) — read-only, harmless.
  • job-scheduled-run: steps[3]'s timeout probe has no retryPolicy, so it cannot show timeouts counting toward retries — add a second probe with a policy; scratch jobs need a writable app copy (copy the app, symlink node_modules, delete dist) — record it as the fixture recipe; objectstack build stops at the first bad job, so each bad shape needs its own build; showcase_health_sweep fires only at 01:00 UTC, so the cron variant rests on a scratch near-term cron.
  • notify-inbox-delivery: showcase_task.assignee is a text field seeded with emails but the notify recipient resolves a USER ID — the reassignment must write the user id; receipts must be read as the recipient (the admin's /data view of sys_notification_receipt is owner-scoped); inbox deliveries do write sys_notification_delivery rows, so acceptance[4]'s "no delivery row" check must compare before/after counts.

Hand-off (items without a judged verdict)

  • Tier 3 deferred by the 18.0 release-sweep plan (not-run): connector-spec-path-no-escape, flow-connector-picker, notification-preference-suppression (all P2).
  • external-schema-browser-ui (P1) — not-run: no whole clause covered by its pin; not hand-driven this run.
  • Partial items' uncovered clauses are listed per item above (not-run rows).

Generated by Claude Code

Activity

  1. objectstack-fleet
    added
    qa-run
    on Oct 19, 2026

  2. objectstack-fleet commented on Oct 19, 2026

    @objectstack-fleet
    objectstack-fleetBot
    on Oct 19, 2026
    ContributorAuthor

    Parked-on: #24579 · this record's checklist-accuracy findings, fixture gaps, partial coverage gaps and not-run handover (wave anchor §2–§4)

    Generated by Claude Code

  3. objectstack-fleet commented on Oct 19, 2026

    @objectstack-fleet
    objectstack-fleetBot
    on Oct 19, 2026
    ContributorAuthor

    Superseded-by: #24615

    Round 2 at ba4afcc re-judged this record's A-class unjudged clauses (all five partials and external-schema-browser-ui now PASS (live legs on local Postgres 16; notify 501 half via a --preset minimal boot)) and carried every other verdict forward to #24615 (Carried-forward: lines there, valid for 5b14b7f). B-class gaps stay parked on #24579.

    Generated by Claude Code

  4. objectstack-fleet
    closed this as not plannedon Oct 19, 2026
  5. os-warren
    added a commit that references this issue on Oct 20, 2026

    docs(qa): 18.0 sweep round 7 — integration-system checklist-accuracy …

    f73b68d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    qa-run

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions