[mchwang]

Closes #164

What changes

• PlanningAgent.probe(vendor, signal) (runner/planning.ts) is the planning check. It runs one planning invocation in planning's own worker and in the same lane D profile a real request gets: read, list and search only, vendor-only egress, and a mounted schema. It runs on the review's code at the snapshot's head, with a real request's budget, because a process's first job may build the agent image. It asks for the schema's own answer, {"ok": true}, and passes only on exactly that (isPlanningProbeAnswer). So it covers:
  • sign-in: the credential, and Anthropic accepting it;
  • a schema-constrained answer;
  • the read-only planning profile starting and returning output.
• PlanningCheck (runner/planning.ts) controls admission:
  • start() runs the check for the vendor chosen in Settings when the server starts. It runs nothing when no vendor is chosen.
  • admit(signal) waits at most 30 s in all for the check that covers the vendor chosen now. All waiting requests share that check, and a request that stops waiting leaves it running. A check still running after the wait gets a refusal: "Planning is unavailable: codeboost is still checking the planning agent."
  • A failed check refuses requests for a minute with "Planning is unavailable: codeboost's check of the planning agent failed; it checks again on the next request after a minute. The check said: ". The reason comes last and unchanged, because some reasons end with a command to copy. The next request after the minute checks again.
  • The first request after the vendor changes starts that vendor's check and cancels the check it replaces.
  • assertPassed() runs again inside the server's recorded action, through the description's validate(). So a vendor chosen during the issue read never runs unchecked. Because that action runs inside the Store's transaction, assertPassed starts no check there. It refuses, and the needed check starts on the next turn.
  • The wrapped provider.invoke checks once more when it is invoked. This is defence in depth: E3 invokes it in a microtask right after the recorded action, where Settings cannot change, but a later caller might. It rejects rather than throws, and E3 then settles the request as failed with the refusal.
  • A check that has not settled within a real request's budget plus a minute fails and is cancelled, so the recheck path applies to it.
  • A reason that holds until codeboost restarts gets no promise of a recheck, and says only a restart can change it. That covers an abandoned worker, cleanup codeboost could not confirm ("off until codeboost restarts"), a database file with other hard links, and a missing CLAUDE_CODE_OAUTH_TOKEN, because codeboost's environment is fixed while it runs. Codex is told to choose another agent, which the next planning request then checks. After close, the refusal says planning is stopping. It matches only codeboost's own text, never the vendor's output.
  • A request whose wait ends without a verdict is refused as such, never recorded as a failed GitHub read. An aborted request reads "stopped waiting"; a Store that cannot be read reads "could not check".
  • close() cancels any running check and waits up to 20 s, including for a timed-out check's probe still in the worker, before the worker closes.
• productionPlanning (web/planning.ts) starts the check at setup. describe awaits admit before the issue is read or a planning request is recorded. The server already records a GuardRefusal as a 409 and starts no agent, so web/server.ts is unchanged.

Interpretation

• "Read-only access" means the read-only planning profile. The check runs in exactly that profile. It does not ask the agent to read a file, because nothing can confirm the agent's account of what it read.
• "A clear 'planning unavailable' state" is shown as the refusal of a planning request: a 409 whose message starts "Planning is unavailable:" and names the reason. Showing it in the review view before a click would need web/server.ts and web/public/*, which other work owns right now. That is follow-up Show the planning check's state in the review view #198, which also covers the web/server.ts limits listed below.
• Limits, documented:
  • A passed check is kept until a check for another vendor replaces it. A worker abandoned later, cleanup that does not settle later (the check's own run included, when its answer still came back), or a token revoked later, shows as a failed request, not as "Planning is unavailable".
  • A run whose cleanup did not settle gets the recheck promise, because its storage is retried before the next run. If planning did go off, the recheck reports that with the restart wording.
  • A refusal recorded when a request stopped waiting, or when the Store could not be read during the wait, is replayed if the browser resends the same action. Keeping a Store error out of the record needs web/server.ts (Show the planning check's state in the review view #198).
  • A check that runs before the runner's setup has set the runner-owned repository fails on a task with runner commits ("which is not configured"). It keeps the recheck wording, because the recheck a minute later passes once the runner's setup has finished. Without a runner block it fails every time, as every planning request would.
  • A recheck joins the worker's single startup recovery, so a recovery that never ends fails each recheck until codeboost restarts.
  • A check deferred from the Store's transaction can start during shutdown, before planning closes. Close then waits for it, within its bound. Stopping it earlier needs web/server.ts (Show the planning check's state in the review view #198).
• Startup. The check starts before the server listens. If startup fails after planning's setup, the CLI exits, and the next start's planning recovery removes what the check left. Another caller of startServer is left with the check running, because startServer does not close planning when a later startup step fails. Fixing that needs web/server.ts, so it is part of Show the planning check's state in the review view #198.

Tests

• test/planning-production.test.ts, "the planning check (Probe the planning profile at runner startup #164)", 39 tests. Each uses a stand-in agent whose probe passes, fails, throws or never settles. They cover:
  • the start at setup;
  • one check for every request;
  • refusing with the reason before the issue read;
  • rechecking after the wait;
  • one shared check, and an abort that leaves it running;
  • the 30 s wait, and admitting once the check passes;
  • the defaults through productionPlanning, with nothing overridden: a 30 s wait, a check failing after the planning budget and a minute, and a 20 s close bound;
  • no vendor, a vendor change, and a change while a request waited, including the agent cleared after the wait ran out;
  • a vendor chosen during the issue read: refused in validate, checked on the next turn, refused as still checking while that check runs, then admitted;
  • a probe that throws;
  • cancelling a replaced check;
  • close cancelling the check and waiting for it, close's bounded wait, no check starting after close (one deferred before it included), nothing admitted after close even under a passed check, and a request waiting at close told at once that planning is stopping, whether its check's probe never stops or then passes;
  • one deadline bounding a request's whole wait across a vendor change;
  • a check that does not settle in time failing and being cancelled, staying failed if its probe answers late, then rechecked;
  • the restart wording for every restart-only reason, each built by the code that gives it (a missing sign-in, an abandoned worker, unconfirmed cleanup, a database with hard links), and none from vendor output;
  • a Store that cannot be read once a wait has begun;
  • the stopping refusal after close, and the Codex wording;
  • close waiting for a timed-out check's probe;
  • a request still waiting after its check was replaced and the vendor changed back, moving to the newer check at once;
  • the deferred check never throwing when the Store cannot be read;
  • many requests waiting on one check without Node's listener warning;
  • a multi-line reason kept whole at the end of the refusal;
  • the provider refusing to invoke an agent whose check has not passed;
  • through the server: a 409, no request recorded, no agent called.
• test/planning-worker.test.ts: the check runs as a planning job in planning's worker, and the stub answers only the check's prompt and schema. It fails without a sign-in, and with Codex before any lock or Docker work. A wrong answer fails it, and only the exact answer passes.
• test/agent-planning.test.ts (real Docker): the check against Anthropic with an invalid token fails with the vendor's 401 and leaves nothing behind.
• Live probe (test/agent-planning.test.ts, "passes the planning check with a real sign-in (live Claude)", added to scripts/live-probes.ts and the Live probes workflow): the check's own prompt and schema, answered by the real Claude service.
  • Started by hand on head d2b4a94, an intermediate checkpoint: run 39533098077 passed all 5 live Claude probes, the new one in 7.1 s.
  • Re-run on head e237aab, an intermediate checkpoint, which includes Sweep leftover path identity probes in startup recovery (#181) #192's recovery changes: run 39534630505 passed all 5 live Claude probes.
  • Re-run on head 8f08e4a, an intermediate checkpoint, after the probe also checks for Docker leftovers: run 39536026699 passed all 5 live Claude probes.
  • Re-run on head 5792621, an intermediate checkpoint, after the answer check became a whole-text match: run 39541887939 passed all 5 live Claude probes, the planning check in 9.1 s.
  • Re-run on head 4cc95fb, an intermediate checkpoint after the merge of main at e46cb68, so the probe runs with Enforce a pre-spawn model budget for planning invocations #163's --max-turns 50 and input budget: run 39550815257 passed all 5 live Claude probes. Later commits change only PlanningCheck, its tests and docs, not PlanningAgent.probe or the answer check.
  • Validated head 6888587 (base e46cb68): run 39558291953 passed all 5 live Claude probes; CI test and real-docker passed on the same head.
• test/browser/plans-production.spec.ts: the lane D stand-in gets a probe to fit the type. The spec chooses no vendor, so the check checks nothing there; it passes locally.

Review record

All rounds were independent-agent reviews: a fresh general-purpose subagent with no author context, reading the full origin/main...HEAD diff each time, never only the fixes. Each round's fixes are one commit whose message starts "Review round N" and lists what changed.

Rounds	Heads reviewed	Findings	What changed
1–7	1a81c20/ce80552 → 02e32b9	blockers and majors: a check started inside the Store transaction, replaced checks left running, unbounded waits, the reason cut or reordered	check started outside the transaction; replaced checks cancelled; one deadline per wait; reason last and whole; live probe added (d2b4a94)
8–20	d2b4a94 → de7c8f5	minors: recovery and shutdown wording, restart-only wording matched on vendor output, the answer check too loose, close not waiting for timed-out probes	UNTIL_RESTART on codeboost's own text only; exact JSON answer; close waits for probes; follow-up #198 filed for server-side limits
21–28	ccf6bd1 → e0e46cb	minors and nits: restart wording, a cleared agent mid-wait, test reasons not built by the code that gives them, Codex stand-ins that pass	reasons built from their sources; Codex stand-ins fail as the real probe does; wrapped invoke's forwarding tested
29–35	4cc95fb → 0472de6	minors: defaults not pinned, admission after close (passed check, no vendor, waiting request), close not waking waiters	fake-clock defaults test; close refuses first everywhere and wakes waiters; tests fail without each fix (checked by removing it)
36	base e46cb68 / head 6888587	none: CLEAN	—

Declined, with reasons (each repeated to later rounds so they were not re-reported):

• The abandoned worker's text ("The first planning request after the restart removes them") is kept: the startup check is a planning request through that worker.
• A refusal quotes the check's reason as given, after "The check said:", including planningOutput's request wording: its sign-in and usage advice applies, and the one-turn probe cannot hit the turn limit.
• checked() does not close planning after each test: close waits up to 20 s for never-settling probes, and the check's timers are unref'd.
• A worker that never ends a job keeps one timed-out probe per recheck window for close to wait on; close's 20 s bound holds (commented).
• Skipping the startup check when a runner is configured: the minute of "not configured" refusals is the accepted limit above.

Merges of main at e46cb68 (#196, #163's planning budget) resolved one table-row conflict in plan-library-alignment.md keeping both rows; the review rounds after it read the merged diff.

Clean independent-agent pass authorizing the Copilot request: round 36, base e46cb68, head 6888587.

Copilot review on head 6888587 (review 5596165142): approval recommended, no open issues, no inline comments.

Review-lesson audit

• Fakes and stand-ins that did not answer as the real code does (Codex probe passing, made-up reasons, the browser stand-in's comment): covered by the existing rule on fakes ("A fake of an external tool … must give … each part of the answer the code reads") and the rule that a test's setup must leave the state the production path would.
• Behavioural claims in comments, docs and the PR body that were false on one path (shutdown timing, "after close nothing is admitted", counts): covered by "Treat every behavioural claim … as something to verify".
• Tests that could not fail (defaults not pinned, a listener limit Node never warns about): covered by "The Design system and plan format #1 Rule of E2E Tests" and the claim rule.
• Evidence without labels (live-probe heads): covered by "In evidence records, label cited commits".
• Wording and layout nits: one-off; no rule.

No new rule is needed, so no AGENTS.md or CLAUDE.md change is in this branch.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

No blocking issues remain in the reviewed admission lifecycle, production wiring, or regression coverage; deferred server and UI work is explicitly tracked in #198.

No open issues 🎉

What changed in this PR

Adds a startup planning-agent check and gates planning requests on its result, addressing #164. Review-view status and server lifecycle follow-ups remain in #198.

Changes:

• Probes the configured agent through planning’s worker and read-only profile.
• Adds bounded admission waits, retries, cancellation, and shutdown handling.
• Adds regression coverage and a live Claude probe.

File	Description
web/​planning.ts	Starts the check and guards planning requests.
test/​planning-worker.test.ts	Tests probe inputs, credentials, and exact answers.
test/​planning-production.test.ts	Tests admission, retries, races, and shutdown.
test/​fixtures/​question-worker-stub.ts	Supplies controlled probe answers.
test/​browser/​plans-production.spec.ts	Adds the probe method to the stand-in.
test/​agent-planning.test.ts	Tests real authentication and cleanup.
scripts/​live-probes.ts	Registers the planning live probe.
runner/​question-leftovers.ts	Clarifies recovery timing in comments.
runner/​question-agent.ts	Extracts shared abandoned-worker wording.
runner/​planning.ts	Implements the probe and admission lifecycle.
docs/​implementation/​runner-lifecycle.md	Documents recovery and shutdown timing.
docs/​implementation/​planning-acceptance.md	Documents check behavior and limitations.
docs/​implementation/​plan-library-alignment.md	Records #164 implementation and follow-up.
docs/​implementation/​ci.md	Documents the fifth live probe.
docs/​implementation/​agent-isolation.md	Adds planning-check validation evidence.
.github/​workflows/​live-probes.yml	Includes the planning live-probe test file.

_{Review effort: Balanced}