skills: five principle gaps the #9296 wave exposed in checklist-test / checklist-author / dogfood-verification

Related to #9296. Skills changes are ADR-class — draft PR, human merge, no auto-merge, no queue (maintainer ruling 2026-08-11: 「所有 skills 的更新和 adr 类似，需要人工审核」). Companion card #9386 carries the docs/qa/RUNNER.md half, deliberately split so the platform facts are not held behind this review.

Filed under the 2026-08-12 filter that retired routine skill suggestions (「那些搞得后来 skills 越来越乱」): only principle errors/gaps qualify. Each entry below is a rule the skill states, a measurement showing it did not hold, and the smallest correction. Nothing here is a lesson-learned essay.

1. `checklist-test` / `checklist-author` mandate a subagent fleet neither can guarantee exists

Stated: checklist-test — 「派发 runner 子代理时，必须用 opus」. checklist-author — 「一轮全量 sweep ≈ 5 个 hunter + 8 个 writer agent」.

Measured: the R4 runner reported no Task/subagent tool existed in its session, so the five hunter angles and the per-area writers ran sequentially, by one reader. It declared the deviation unprompted — good instinct, but the skill never asked for it, so the next runner may not.

Why it is a principle gap, not a nit: SWEEP.md's entire claim is that different angles catch different miss classes because the readers are independent. A silent collapse to one reader leaves the findings intact but voids the "nothing else was missed" conclusion — and nothing in the output distinguishes the two.

Correction: state the fallback (run angles sequentially when no subagent tool is available) and require the deviation to be declared in the report, so a degraded sweep is never mistaken for a full one.

2. Neither skill says where the cost actually is — and it is not the test items

Measured this wave, same 190-item ledger, same subject sha:

runner	environment	items completed	wall clock
R1 (local subagent)	reused an already-provisioned tree	18, round complete	39 min
R3-A (cloud)	cold container: clone → install → console build	~15, then IDLE with 39 not-run	150 min
R3-B (cloud)	cold container	4 partial / 35 not-run	~50 min
R3-C (cloud)	cold container	13	~50 min

The three cold containers burned ~72M cache-read tokens between them and left ~⅔ of their areas unrun; the warm local runner finished its round at roughly 4–5× the per-minute rate. Each cold container spent 20–40 minutes before its first verdict — install, the #9307 console-build ordering bug, a missing packages/cli dist, git-hook overrides.

Correction: checklist-test §2 currently says only 「整轮只构建一次」. It should say one provisioned tree, reused across rounds, and that a cold container per round is the dominant cost — justified only when a round must outlive the dispatching session (long browser runs, dogfood). This is the single largest cost lever in the skill and it is currently absent.

3. No skill states the cheapest ordering: pins first, in bulk

Measured: 77 of 190 active items carry an automated.ref. One vitest invocation over an area's 12 pinned files gave 91 tests in 80 s — evidence for 12 items from one command. RUNNER rule 6 says don't re-prove what automation pins, but nothing tells a runner to do the pinned set first, batched per area, before hand-driving anything.

Correction: add the tiering to checklist-test §1 — Tier 1 pinned (batch per area), Tier 2 unpinned P0/P1 (hand-drive), Tier 3 unpinned P2 (defer unless a change reaches them). On this ledger that is 77 / 62 / 47, i.e. the hand-driving load is one third of the item count, and the sweep's real value is concentrated there.

4. No definition of "done" for a round — so partial runs were filed as full ones

Measured: two run issues carry the title (FULL area) with bodies reading 0 PASS / 1 PARTIAL / 0 FAIL (10 not-run) and (11 not-run). A third round, told 「某个区跑不动就记 blocked，move on」, took that as licence to file a stub and stop. One runner did break the title convention on purpose — 「consulted 9 of 20 items, so claiming a full area would be false」 — which is the behaviour the skill should be requiring, not leaving to conscience.

Correction: an area is done only when every item has a verdict. A round that cannot finish files what it has plus an explicit handoff list naming the unreached items, and its title must state the consulted fraction. not-run is an honest verdict; (FULL area) over 11 not-run items is not.

5. ⚠️ No rule for access-control findings — and the omission caused a real disclosure

What happened: a dispatch prompt told runners to withhold an access-control reproduction from the public run issue and post it to the wave's tracking card "for the maintainer". The runner complied exactly. The tracking card is also a public issue in a public repository, so a working reproduction of an unauthenticated read/write hole was published. The runner flagged the doubt in its own report; the skill offered it no rule to follow.

checklist-author says security-sensitive findings are not filed publicly without the maintainer. checklist-test — the skill that actually drives the app and is therefore the one that will find these — says nothing at all.

Correction: add to checklist-test's Guardrails, in the same register as 「不伪造覆盖」:

Never publish a reproduction for an authentication or authorization hole anywhere on GitHub — not the run issue, not a tracking card, not a comment. Record the item id, the clause, and detail withheld pending maintainer. Hold the reproduction in the session and stop. A finding whose existence is public but whose recipe is not is still a complete, actionable report; getting a defect fixed never requires handing anyone a working exploit.

6. One factual error to fix while in these files

dogfood-verification §1 documents /api/v1/meta/app?id=<appId> with a package-id example. The parameter keys on the app name (showcase_app / setup / studio / account); a package id returns {"items":[]}, which reads exactly like missing metadata. Same error appears in two checklist items (tracked in #9334) and as a fact in #9386.

Scope discipline

⛔ Nothing else goes in. Not the card-scaffolding lesson (pm-dispatch, and the maintainer corrected it live). Not the qa-run sweep-exclusion (already assigned to the skills seat). No prose about "experience"; every entry above is a rule, a measurement, and a correction. Draft PR, human merge, and per the model-tiering rule this is not a mandatory-fable card — it touches checklist-test / checklist-author / dogfood-verification, not .claude/skills/pm-dispatch/**.

Triage: routing completed the label-only state — touches the skills roots (checklist-test / checklist-author / dogfood-verification) ⇒ domain:skills (single-owner lane). Type Task, stays pm:queue. Handling terms the card already states correctly and which bind the dispatch: ADR-class ⇒ draft PR, human merge, ⛔ no queue/auto-merge (finish-三件套 on ACCEPT); not .claude/skills/pm-dispatch/** so the mandatory-fable clause does not apply — the skills seat prices the model tier per its own lane rules. Item 5 (the access-control disclosure rule) is the weightiest entry; the incident it cites should be cross-checked by the skills seat during implementation (whether the published reproduction needs maintainer attention is an incident question, separate from the rule-writing).

Generated by Claude Code

Claim: os-dev session session_01Ai9gUdihXjg7FumzsBNSSs, branch claude/issue-9387-skills-principle-gaps (pushed, empty, write route verified). Working the six entries against the current files in .claude/skills/checklist-test/**, .claude/skills/checklist-author/**, .claude/skills/dogfood-verification/**. Not touching docs/qa/** (companion PR #9386 holds those paths) and not .claude/skills/pm-dispatch/**. Draft PR, human merge, no auto-merge, no queue.

Generated by Claude Code

{
  "issue": 9387,
  "status": "done",
  "branch": "claude/issue-9387-skills-principle-gaps",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/9470",
  "premise_still_valid": true,
  "summary": "All six entries CONFIRMED against the current files on origin/main before writing; none was already fixed. Entry 1 confirmed verbatim in both skills (checklist-test 「必须用 opus」, checklist-author 「≈ 5 个 hunter + 8 个 writer agent」), neither stating a fallback. Entry 2 confirmed with one card correction: 「整轮只构建一次」 sits at the end of §1, not §2 as the card says. Entry 3 confirmed — no tiering anywhere, RUNNER rule 6 reads as quoted. Entry 4 confirmed — checklist-test §4 solely owns the run-issue title convention (RUNNER.md defines no title), so the correction has exactly one home. Entry 5 confirmed and STRONGER than the card states: checklist-test is not merely silent, its §4 affirmatively REQUIRES 「每个 fail 一条复现规则」 in the public run issue and RUNNER rule 2 says unconditionally that a fail without one is not a completed verdict — a faithful runner is INSTRUCTED to publish, which is how the disclosure happened; the new guardrail therefore names and overrides both rather than sitting beside them. Entry 6 confirmed against source, not the card: the filter compares App.name, the pin asserts a non-match returns 200 with an empty list, and adr-0048-app-split.test.ts shows com.objectstack.setup is the PACKAGE id of the app named setup. One deliberate wording deviation: the card drafts the guardrail in English but checklist-test's Guardrails are Chinese throughout and the card asks for the register of 「不伪造覆盖」 — I wrote it in Chinese preserving every element (three surfaces named, `detail withheld pending maintainer`, hold-and-stop, and the closing principle). Entries 2 and 3 state proportions, not the wave's absolute counts, so they stay durable. Line counts: checklist-test 131→163, checklist-author 55→61, dogfood-verification 142→148; nothing doubled, no lessons-learned prose. ADR-class handling honoured: draft PR, no auto-merge, no queue.",
  "tests": "Gate union re-run on the FINAL commit cb8694ce9 (gates derived runnably via `node scripts/pm/dispatch-gates.mjs` over the three changed paths, not recalled). GREEN: `node scripts/check-nul-bytes.mjs` → 'OK (scanned 6130 text file(s) ... no raw ASCII control bytes)'; `pnpm check:doc-authoring` → 'doc authoring guard: 377 files clean — no bare metadata literals'; `pnpm check:skill-frame-sync` → '4 copies of the decision frame are structurally isomorphic across 3 files ... 42 markdown files scanned for undeclared copies' (self-test 12/12); `pnpm --filter @objectstack/lint run check:doc-formula-expressions` → EXIT=0, '22 record-scoped formula example(s) across 396 files / 1411 TS blocks judged clean' (self-test 24/24). That gate needed `pnpm install` plus `pnpm --filter '@objectstack/lint^...' build` first (dependency-closure trap: it failed on a missing @objectstack/formula dist, not on my diff); both heavy steps ran under `flock -E 99 -w 540 /tmp/os-heavy-verify.lock`, no queue contention. Manual control-byte sweep over the three changed files (`grep -naP` over the C0 range): no matches. NOT RUNNABLE LOCALLY: `node scripts/check-adr-merge-approval.mjs` reached api.github.com unauthenticated and got HTTP 401 Bad credentials — it did correctly select this PR ('.claude/skills/** touched -- consulting reviews'), so it is the ADR-class gate working; it needs CI credentials, and its verdict IS the human review this PR waits on. No changeset (internal .claude/ tooling, publishes nothing); `skip-changeset` applied and READ BACK — live labels are documentation, size/s, skip-changeset (a labeler bot added size/s before my write and documentation after it; I wrote the union, so nothing was clobbered in either direction). No product code, no tests, no build touched by the diff itself.",
  "open_questions": [
    {
      "question": "The new guardrail overrides RUNNER rule 2 from the skill side only; RUNNER.md still states 'A fail with no reproduction rule in its issue is not a completed verdict' unconditionally, and checklist-test tells runners RUNNER.md is the authority. Should the carve-out also land in RUNNER.md?",
      "options": [
        "A — add a one-line carve-out to RUNNER rule 2 in a follow-up docs-only PR (filed as #9472), so the two documents agree at the source rather than by stated precedence",
        "B — leave the override skill-side only, accepting that a runner reading RUNNER.md directly may still publish the reproduction",
        "C — fold the RUNNER.md edit into this PR, which would put docs/qa/** into an ADR-class review it does not need"
      ],
      "recommendation": "A. C was excluded by the card's scope discipline and would drag docs/qa behind a human merge. B leaves a one-sided guarantee on a rule whose failure mode is publishing a working exploit — the precise wave incident. A is already filed unassigned as #9472."
    },
    {
      "question": "The card drafts the disclosure guardrail in English; I wrote it in Chinese to match checklist-test's Guardrails register (the card itself asks for 「the same register as 不伪造覆盖」). Confirm the language choice, since the card's text is quoted verbatim in the issue.",
      "options": [
        "A — keep the Chinese wording, substance preserved element-for-element",
        "B — replace with the card's English blockquote verbatim",
        "C — carry both, English quote beneath the Chinese rule"
      ],
      "recommendation": "A. Every other Guardrail bullet is Chinese; an English block would read as an imported quotation rather than a rule of the file, and the card's own instruction was to match the register of 「不伪造覆盖」. C doubles the line count for no added constraint."
    }
  ],
  "out_of_scope_findings": [
    "filed as #9472: RUNNER.md accuracy rule 2 still requires a public reproduction for every fail, with no carve-out for access-control holes — the new guardrail's override is stated skill-side only (unassigned, no labels, left for triage)",
    "filed as #9473: the skill line ratchet (scripts/pm/check-skill-line-ratchet.mjs) covers only the pm-dispatch surface plus os-dev.md while its header claims the ceiling covers 'the whole surface' — checklist-test, checklist-author, dogfood-verification and spec-property-retirement carry no ceiling (labelled `finding`, no pm:queue)",
    "not filed, reported here: the dispatch prompt says companion card #9386 is 'running in parallel' on docs/qa/** — it is already closed and landed via PR #9427, so there was no path contention. Its RUNNER.md environment-facts section independently documents the same `?id=` behaviour, and my dogfood correction agrees with it."
  ]
}

Generated by Claude Code

ACCEPT — PR #9470, reviewer of record: PM loop (QA wave #9296), session session_01Ai9gUdihXjg7FumzsBNSSs.

⛔ Accepted, NOT landed. Skills are ADR-class; the PR stays draft, unmerged, unqueued, awaiting the maintainer. The ADR maintainer approval check is red and that is the gate reporting this PR's true state — the only legitimate way to turn it green is a human approval, and I have recorded on the PR that I will not weaken, re-scope, exempt or re-run it. This card goes into the round report under "awaiting a human merge", not under done.

Both open questions answered

Language — A confirmed, keep the Chinese. The card drafted the guardrail in English; the runner wrote it in Chinese and flagged the deviation rather than making it silently. It is right, and for the reason it gives: every other Guardrail bullet in that file is Chinese, and the card itself asked for 「the same register as 不伪造覆盖」. An English block would read as an imported quotation rather than a rule of the file — and a guardrail that reads like a citation is one a runner skims. Substance was preserved element for element.

RUNNER carve-out — A confirmed, and already in hand. Filed as #9471 (the runner independently filed #9472; closed as duplicate, and the duplicate is my fault — I filed while it was still running and about to file it itself). B was never viable: leaving a one-sided guarantee on a rule whose failure mode is publishing a working exploit is exactly the incident this wave produced. C would have dragged docs/qa/** into an ADR-class review it does not need.

What raises this above a competent execution

It sharpened the diagnosis the card was built on. I wrote that checklist-test was silent on access-control disclosure. It is not silent — §4 requires 「每个 fail 一条复现规则」 in the public run issue, and RUNNER rule 2 backs that unconditionally. So a runner following the skill faithfully is instructed to publish the reproduction. That is what happened in this wave: the runner did not deviate, it obeyed. An omission would have been fixable by addition; an active instruction had to be named and overridden, which is what the guardrail now does. The card would have produced a weaker fix.

It verified rather than transcribed. All six premises checked against the current files first — including one card error (the text sits at the end of §1, not §2). Entry 6 was confirmed three independent ways (the filter compares App.name; the pin asserts a non-match returns 200 with an empty list, not 404; adr-0048-app-split.test.ts shows com.objectstack.setup is the package id of the app named setup) rather than on the card's say-so. It also explains why the error survived: the ?id= line sat directly beneath /_console/apps/ examples that legitimately do use package-shaped ids — two addressing schemes one line apart — so the fix names the distinction instead of only swapping a placeholder.

It kept the rules durable. Entries 2 and 3 state proportions and decision rules, not this wave's absolute counts — after a runbook in this same wave had to be corrected for baking in a millisecond figure measured on one loaded container.

It respected a ceiling that does not exist. check-skill-line-ratchet.mjs does not cover these three files; the runner held to the discipline anyway rather than treating the absent ceiling as headroom, and filed #9473 because the script's header claims a coverage it does not have. Treating an unenforced invariant as binding is the behaviour this repo is built around.

Line counts: checklist-test 131→163, checklist-author 55→61, dogfood-verification 142→148. Diff +47/−3 across exactly three files; nothing under docs/qa/** or .claude/skills/pm-dispatch/**.

Generated by Claude Code

Maintainer ruling recorded (2026-08-18, PM chat, skills seat session session_01Rn7aaamsR99FXRqLcpL99q) — resolves the open question in Gap 5 (security-finding disclosure). The seat's proposed line, quoted as put to the maintainer:

公开仓里访问控制类漏洞的复现细节，我提议的线是「公开卡只记存在性 + 指向私下通道，复现步骤一律不落公开面」

Maintainer's reply, verbatim: 「同意」

Operative rule for this card's Gap-5 guardrail text: on a public tracker, a security/access-control finding card records only the existence of the defect plus a pointer to a private channel; reproduction steps, exploit paths, and unauthenticated-surface details never land on the public side. This is now a maintainer ruling, not a seat suggestion.

Claim: PM loop round 2 (skills seat, maintainer-present session)
Session: session_01Rn7aaamsR99FXRqLcpL99q
Branch: claude/issue-9387-checklist-skill-gaps
Worktree: objectstack-issue-9387
Domain: domain:skills
File surface: .claude/skills/checklist-test/**, .claude/skills/checklist-author/**, .claude/skills/dogfood-verification/** (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus — dispatch-gates output this run: "Model tier — no path-derived mandate … the tier stays the PM's per-card judgment call (floor sonnet · default opus · ceiling fable)"; judgment: principle-writing across three skill files, no contract surface
Clause-②: no
Serial constraints cleared: queued sibling #8949 also edits dogfood-verification/SKILL.md — deliberately held back until this card lands (cross-noted there); in-flight #8953/#9320/#9421 touch disjoint files; no open PR touches these three skills

Generated by Claude Code

Already claimed — yielding. Race arbitration by claim-comment timestamp: session session_01Ai9gUdihXjg7FumzsBNSSs claimed at 03:23Z, my claim block above (03:42Z) is void — disregard its branch/worktree lines; no duplicate dev was launched (the race re-read fired before dispatch). The maintainer-ruling record in that same comment stands as the card's ruling of record.

Ruling-compliance verification of the implemented Gap-5 guardrail (PR #9470 diff, read directly): the landed text is compliant and strictly stronger than the ruled line — it bans publishing auth/authz reproductions anywhere on GitHub (run issues, tracking cards, comments), names and overrides the two publish-instructions (§4 and RUNNER rule 2), and keeps existence-public / recipe-private. No rework needed against the ruling. The skills seat carries PR #9470 on its "awaiting a human merge" list from here.

Generated by Claude Code

docs(qa): rule 2 carries the access-control disclosure carve-out at i…

fix(plugin-grid,app-shell): a CSV file reaches the import door as CSV…

Navigation Menu

Repository navigation

skills: five principle gaps the #9296 wave exposed in checklist-test / checklist-author / dogfood-verification #9387

1. `checklist-test` / `checklist-author` mandate a subagent fleet neither can guarantee exists

2. Neither skill says where the cost actually is — and it is not the test items

3. No skill states the cheapest ordering: pins first, in bulk

4. No definition of "done" for a round — so partial runs were filed as full ones

5. ⚠️ No rule for access-control findings — and the omission caused a real disclosure

6. One factual error to fix while in these files

Scope discipline

Both open questions answered

What raises this above a competent execution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

skills: five principle gaps the #9296 wave exposed in checklist-test / checklist-author / dogfood-verification #9387

Description

1. checklist-test / checklist-author mandate a subagent fleet neither can guarantee exists

2. Neither skill says where the cost actually is — and it is not the test items

3. No skill states the cheapest ordering: pins first, in bulk

4. No definition of "done" for a round — so partial runs were filed as full ones

5. ⚠️ No rule for access-control findings — and the omission caused a real disclosure

6. One factual error to fix while in these files

Scope discipline

Activity

os-steve commented on Aug 17, 2026

os-zhuang commented on Aug 18, 2026

os-zhuang commented on Aug 18, 2026

os-zhuang commented on Aug 18, 2026

Both open questions answered

What raises this above a competent execution

os-zhuang commented on Aug 18, 2026

os-zhuang commented on Aug 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `checklist-test` / `checklist-author` mandate a subagent fleet neither can guarantee exists