QA run · integration-system (FULL area) · 92f26f75 · 2026-08-11 · 5 PASS / 5 PARTIAL / 3 FAIL #7690

Full integration-system area run of the checklist-test skill — all 13 items driven against a live showcase across 3 opus subagents, each on its own port + file DB. Text-only per RUNNER.md.

Result: 5 PASS · 5 PARTIAL · 3 FAIL.

⚠️ One security-sensitive finding from this area is NOT described here. It surfaced as an adjacent observation during webhook-lifecycle and was delivered privately as FOLLOW-UPS.md D12, per the maintainer ruling recorded on #7463. It does not change that item's passing verdict.

Environment — framework 92f26f75/f268b2b2 (branch claude/platform-test-checklist-ocwugl) · console bundle 09987b680 (packages/console/dist/.objectui-sha) · showcase app · isolated port + file DB per batch · scratch app copies in /tmp for every mutated-metadata boot (no repo file was written at any point).

✅ PASS — 5 items

• connector-stdio-default-deny (4/4 + both negatives) — the strongest security result of the area, and it is proven by non-spawn, not by an error message. The runner instrumented the fixture to append to a marker file on start: the marker appears on the allowlisted boot and is absent on both deny boots. All four variants covered — policy absent → boot aborts with the feat(connector-mcp): policy-gate declarative stdio transports — default-deny + host allowlist (ADR-0097 security follow-up) #3055/ADR-0024 message; allowlist ['node'] + python3 → refused naming both the command and the configured allowlist (strict equality, no prefix/glob); allowlisted → materializes and dispatches; http transports not subject to the policy (0 declarativeStdio hits, degrades on unreachability instead). The refusal is fatal-at-start, never a degraded husk retrying toward an unauthorized process. Pin re-run 13/13.
• connector-descriptor-audit (3/3 + both negatives) — the inert-descriptor warning names the offender and prescribes both fixes; it is quiet for enabled:false and quiet when a same-name runtime registration exists — proven with a discriminating test, since the same audit pass named the other connector, making the silence attributable. Never fires for provider-bound instances across ~12 boots. Fires in-process on metadata:reloaded with no restart. Pin 9/9.
• webhook-lifecycle (6/6 + all three negatives) — declared webhooks materialize with both spec remaps; all three per-record variants deliver to a local receiver, each keyed by its own event. Bulk-vs-per-record separation proven with two subscribers on the same object: a predicate flow produced exactly 2 aggregate-shaped deliveries (matched:1 / matched:2, no recordId) on the bulk path and zero on the per-record path, with a control create afterwards proving that subscription was alive. HMAC recomputed by hand from the raw body → exact match. Authored timeoutMs enforced; a 404 target goes straight to dead; an unreachable port retries per the backoff ladder with every attempt durable. Webhook triggers undelete and api are declared but never fire #3196 drift-guard verified by inserting legacy-shaped rows directly into the runner's own DB — the enqueuer warns every cycle, naming the bad trigger and that the row "is dead while looking armed in Setup". Audit: several event/subscription/connector enums are schema-only (declared, no runtime consumer) #3197 confirmed: connector-attached webhooks are never read at registration.
• notify-inbox-delivery (5/5 + both negatives) — scoping is server-side (a third member's authenticated listing is empty; sys_inbox_message carries only the assignee's user_id). The 501 capability-unavailable envelope was proven on a real messaging-less boot, not asserted. An unregistered channel dead-letters honestly (status:'dead', error: "channel 'push' not registered") with no inbox row and no receipt.
• notification-preference-suppression (5/5 + both negatives) — suppression is a genuine drop with a located log line and acted=0; the reversal restores delivery. Precedence proven with a control: a user_id='*' row does suppress (a persona with no specific row got nothing) while the user-specific row overrides it. The mandatory-topic bypass was proven by booting a scratch config with MessagingServicePlugin({mandatoryTopics:['task.assigned']}) mounted explicitly — the CLI never sets it — and the row landed despite an enabled:false preference.

🔴 FAIL — 3 items

1. email-template-render — three defects, none of them stale-dist

All three are identical in src, in the dist the server executes, and on origin/main.

(a) A declared i18n bundle collides — only the last locale survives. A stack declaring two emailTemplates with the same name and different locale logs [Registry] Overwriting email_template: <pkg>:<name> and materializes 3 of 4 authored templates; the en-US twin is gone. Root cause: packages/objectql/src/registry.ts (~line 1826) keys items ${packageId}:${baseName} — locale is not part of the key. This contradicts the spec's own statement that "multiple rows with the same name but different locale form an i18n bundle".

(b) A no-locale sendTemplate renders an arbitrary locale. With a bundle present as rows, calling sendTemplate with no locale rendered the zh-CN row — on two consecutive fresh boots — instead of the documented en-US default. Root cause: plugin-email/src/email-plugin.ts (~line 548) templateLoader.load() queries {name} with limit:1 and no ordering when locale is undefined, and email-service.ts (~line 1121) applies the en-US fallback only when preferred was set. Explicit locales resolve correctly (zh-CN exact; fr-FR and en-GB fall back to en-US); language-only zh does not prefix-match.

(c) A runtime metadata write never materializes without a restart. PUT /api/v1/meta/email_template/<name> → 200 and [Registry] Registered email_template, but the row never reaches sys_email_template — and neither of the subscription's own log lines fires (0 occurrences of "materialized from a runtime write", 0 of "runtime email-template sync failed"), so the callback registered at email-plugin.ts ~line 968 never runs, even though its boot line "subscribed to email_template metadata changes" is present. Decisive control: restarting on the same DB materializes it immediately ({seeded:1, skipped:1}) — so the bridge handles the item fine and only the live event path is dead. The plugin's own comment names this as the live path ("Studio saves / PUT /meta land as added/changed events").

The rest of the item passes: rendering with every hole substituted and zero placeholder residue across ~12 sends; a missing required variable throws MISSING_VARIABLES: title with 0 messages reaching the transport while a missing optional one renders empty; TEMPLATE_INACTIVE withdrawal works and is reversible; from/replyTo overrides land with a control proving the default; the authoring-schema aliases all prescribe the right slot (content→bodyHtml, from→fromOverride, title→subject) while a non-alias gets no bogus suggestion; and the raw /email/send door answers 200 / 401 / 400 / 501 correctly, never a 500 for caller-fixable input. #4509 false compliance does not reproduce — an admin edit takes effect on the very next send.

2. job-scheduled-run — job timeouts are not enforced in the durable record

Reproduction rule — declare a job with timeout: 2000 whose handler sleeps 10 s; let it fire; read sys_job_run and sys_job.
Expected run status timeout, counted as a failure (per JobExecutionStatus/#3494).
Actual every row is status: 'success' with duration_ms 10025 / 10003 / 10001 / 10004 — five times the declared timeout — and sys_job reads last_status: success, failure_count: 0.

The scheduler does abandon and retry: a retryPolicy variant fired attempt 2 at timeout+backoff (+2.5 s). Yet both attempts still landed as success rows.

Root cause (read in the running dist, re-checked against the newer src — identical). DbJobAdapter.schedule() passes its recording wrapper into IntervalJobAdapter/CronJobAdapter as the handler, so runWithPolicy's Promise.race timeout guard wraps the recorder instead of the other way round. When the guard rejects, the recorder is still awaiting the un-cancellable handler; when that finally resolves, wrap() writes finishRun(runId,'success').

Why the pin stayed green: the timeout status the adapters do compute lives only in the in-memory JobExecution list that sys_job_run never reads — so interval-job-adapter.test.ts's "enforces a per-attempt timeout and records status 'timeout'" passes against the in-memory adapter while the persisted truth an operator sees says success. The same seam explains sys_job_run.attempt being hardcoded to 1 for every retry (and each retry inflating run_count).

Everything else passes: all three schedule variants verified by their own run row and side-effect tick file (interval to the ms, cron exactly on the minute, once at the declared instant); upsert-by-name holds across restarts; retry backoff measured exactly (1000 ms then 2000 ms); the policy-less probe ran once, confirming the 17.0.0 maxRetries default of 0 (#4661), not the old 3; both skip doors are loud and distinct; runtime job creation is refused 403 NOT_CREATABLE (#4509 door closed).

3. external-datasource-federated-read — the federated read returns nothing

Reproduction rule — on a stock boot: GET /api/v1/datasources/showcase_external/remote-tables → 200 listing customers(7 cols) + orders(7 cols); then GET /api/v1/data/showcase_ext_customer.
Expected the 3 fixture rows. Actual 200 {"records":[],"total":0}; by-id → 404 RECORD_NOT_FOUND. Reproduced on two boots with fresh DBs.

The remote SQLite genuinely holds 3 customers and 4 orders (read directly and via the platform's own introspection route). A filter on a non-existent column also returns 200 empty rather than a driver error — proof the read never reaches the external SQL driver at all.

Suspected root cause (boot ordering). No driver available for object, skipping schema sync {"object":"showcase_ext_customer"} is logged at 09:38:37.014, while Registered driver {"driverName":"showcase_external"} and Auto-connected declared datasources only happen at 09:38:38.413 — so ObjectQLPlugin's federated branch (plugin.ts:1136-1152, the SqlDriver.registerExternalObject call that installs the object→remote-table mapping) never runs, leaving the query path with no binding.

Second, separately actionable hazard found on a control boot where the mapping did register: the platform injects its org-scoping predicate onto a federated remote table that has no such column — select * from \customers` where (`organization_id` = '…' or `organization_id` is null)`.

Also failing (clause 1): the write refusal is correct server-side — ExternalWriteForbiddenError names the datasource, its schemaMode, and both flags that would be required — but reaches the client as a bare 500 INTERNAL_ERROR, indistinguishable from a crash. Nothing was applied: the remote table is byte-identical after both attempts.

🟡 PARTIAL — 5 items

The connector trio, and one cause behind three partials

connector-declarative-boot, connector-degraded-recovery and connector-spec-path-no-escape each pass their substance and each carry the same unproven half.

What is proven: all three provider variants materialize with actions derived exactly from the upstream; all three malformed authoring shapes are refused by the real build gate with located ADR-0097 messages; an unknown provider key aborts boot with rollback, listing the installed providers. A dead upstream degrades without taking the boot down, stays visible as a husk with its reason, fails dispatch fast (31 ms) quoting that reason verbatim, and recovers restart-free — proven from one continuous boot log (zero shutdown lines) after restoring the upstream file. Backoff doubling was measured exactly (5.07 / 10.11 / 20.19 s). Operational and configuration faults are correctly split. Path confinement was tested with a canary: the out-of-root target was a valid OpenAPI document whose operationId would have materialized a working action had the guard leaked — across absolute, ../ and embedded ../../../../ probes, the connector never materialized and the canary appears 0 times in every boot log; the check is on the resolved path, and a path resolving back inside the root is accepted.

The shared gap: the dev metadata reload does not re-ingest connector items. reconcileDeclaredConnectors reads ql.registry.listItems('connector'), and the artifact reload re-ingests object definitions but not connector items — so a connector edit on disk fires a reload with no reconcile, no teardown, no re-materialize. Not scored as a fail: the default os dev restarts the serve child on recompile, so the user-visible loop does apply the change, and the production trigger (a Studio package publish) was not drivable here. Flagged for triage. Also unproven: the 300 s backoff ceiling (the runner stopped at the 40 s cycle rather than wait ~10 min) and "a config edit resets the backoff" — a reload does force an early retry but advances the backoff rather than resetting it, matching the source, where only a signature change resets.

datasource-admin-lifecycle

Secret handling is exemplary and worth calling out next to D12: hasSecret is a boolean with no value in list or detail, and a byte-level scan of every table and column of the running DB found no occurrence of the inline secret — it is split out server-side and never lands in the persisted draft. Refusals attribute to their own service (DATASOURCE_ADMIN_ERROR vs EXTERNAL_DATASOURCE_ERROR), so the #4249 mis-attribution is absent.

partial solely because clause 4 is blocked(fixture) exactly as the item's own knownGaps pre-authorises — os dev wires ExternalDatasourceServicePlugin by default, so the 503 cannot be provoked. Confirms D6: the admin CRUD routes actually mounted carry no route-ledger entry (the ledger's five datasource entries are all federation ones, and the ledgered /external/tables spelling differs from the live /remote-tables) — the tranche-3 gap stands. Observation: a draft naming an unknown driver is accepted 201 with status:'error' rather than refused — health reporting rather than validation.

flow-connector-picker

The picker follows the runtime registry, not the metadata list: its five options diff name-by-name against GET /automation/connectors, and it correctly does not offer showcase_erp_catalog — the catalog descriptor that GET /meta/connector does list — so it cannot wire an undispatchable node. Provenance annotation is exact. Actions are derived end-to-end.

partial: action input schemas are not surfaced. After committing an action, the Input section stays a generic untyped key/value repeater, with no field derived from the descriptor's inputSchema. git grep inputSchema over app-shell at the served pin and on origin/main hits only default-schemas.ts — nothing consumes it, so this is not fixed upstream.

Notes worth keeping

1. Seed-data trap in the notify path: the stock seed assigns ada@/linus@/sam@example.com, which are not sys_user rows and resolve verbatim to non-users — so a stock reassignment lands an inbox row nobody can read. The runner signed up real personas and used their emails instead.
2. POST /api/v1/data/sys_account now answers 405 (create not allowed), so the previously-used recipe for turning a seeded demo persona into a real login is closed on this build.
3. A flow authored over the metadata API is not editable in Studio — with or without ?package=, it opens behind "This flow is provided by an installed package, so it is read-only at runtime", because ResourceEditPage treats an item as artifact-backed when layered.code != null && _packageId !== 'sys_metadata' and flow declares allowOrgOverride:false. The editable path is the console's own create page.
4. Checklist slip — email-template-render clause 7 names a {to, subject, bodyHtml} message body; the wire keys are html/text (bodyHtml is the template field). The literal shape the clause names is refused 400 "at least one of text or html is required".
5. Predicate (multi:true) writes are unreachable over REST by design (deleteManyData 把请求体的 options 展开在 where 之后 —— body 可覆盖 id 谓词,把"删这几条"放大成"删调用者可见的全部" #3897 strips options.multi from the batch routes), so the bulk-webhook clause needs a flow update_record/delete_record node with multi:true fired through the api trigger — recorded so the next sweep does not re-derive it.
6. Notification observation: the ['push'] probe's flow-run summary reports status:'success', acted:1 while nothing was delivered. The dead-letter is visible in sys_notification_delivery but not in the run summary.