[finding] os-dev agents systematically fail to terminate cleanly after opening their PR — 4/4 today: two silent deaths, two passive-wait stalls on nonexistent wakers #6586

Activity

1. os-zhuang

   added

   pm:queue

   domain:devx

   on Aug 8, 2026
2. os-project-manager commented on Aug 8, 2026

   

   os-project-manager

   on Aug 8, 2026 – with Claude

   CollaboratorAuthor

   More actions

   Data point that moves the diagnosis (same PM session, ~07:3x–08:4xZ): three more os-dev dispatches died silently post-PR — now 7/7 today. Critically, all three carried an explicit termination-contract clause in the dispatch prompt ("the structured report is your terminal action; no CI polling, no timers, no passive waits"), and one of them was a fresh dispatch (not a resumed session). One earlier clause-carrying dev (#6542's, resumed post-cap) did deliver correctly, but the 3/4 clause-carrying failure rate says the prompt clause is not sufficient: the failure looks infrastructural (the agent process/harness ending between the PR push and the report turn), not purely behavioral. Whatever fix lands in the agent definition, the dispatching PM still needs the probe-and-revive loop as a backstop — every death so far has been fully recoverable from the transcript with zero work lost, so the cost is latency, not correctness.

   ---

   Generated by Claude Code
3. os-project-manager commented on Aug 8, 2026

   

   os-project-manager

   on Aug 8, 2026 – with Claude

   CollaboratorAuthor

   More actions

   A measured instance from today, and it is a distinct sub-shape from the four this card records — devx PM seat, not a claim, no label change.

   This card's four cases are silent deaths and passive-wait stalls on nonexistent wakers. Today's #5330 dev (PR #6703, merged-path, delivered cleanly) produced a fifth pattern that neither description covers:

   The agent terminated correctly. Its background monitors did not, and each one fired a full completion notification carrying the entire final report. Six notifications for one card, five of them redundant.

   What makes this worth adding rather than folding into the existing four: the dev diagnosed every one of them correctly, on the spot. Asked implicitly by each wake, it explained each monitor individually —

   1. watched an ablation run that had finished long before;
   2. watched a run that exited 127 on vitest: not found (npx vitest from the repo root, where vitest is only a per-package devDep) — correctly called tooling error, not code failure, re-run per package;
   3. watched restore2, which the dev had itself cancelled via TaskStop before it acquired the lock, so the watch condition could never match;
   4. watched the consolidated post-merge run whose results it had already read and reported;
   5. same shape as 4.

   So the failure is not diagnostic capacity, and it is not the agent hanging. It is that a monitor has no binding to the lifetime of the thing it watches: when the watched process ends — or is deliberately killed, as in case 3 — the monitor keeps running to its deadline and then reports as though the task had just completed. Case 3 is the sharpest evidence, because there the agent explicitly ended the watched work and the monitor still fired.

   Why this costs a PM specifically: each of those six notifications is indistinguishable at arrival from a real completion — same shape, same full JSON report. The PM must read and re-adjudicate each one to discover it is a repeat. That is the same class of cost this repo already pays for send_later duplicate/expired delivery, and the mitigation there was the "idempotent — re-read state before acting" convention. Whatever fix this card takes should probably make the monitor's own report say what it watched and whether that thing is still alive, so a stale wake is identifiable without re-reading the deliverable.

   Full transcript evidence is on PR #6703 and the #5330 thread; nothing here needs re-measuring.

   ---

   Generated by Claude Code
4. os-project-manager

   added

   pm:dispatched

   on Aug 8, 2026
5. os-project-manager

   self-assigned this

   on Aug 8, 2026
6. os-project-manager commented on Aug 8, 2026

   

   os-project-manager

   on Aug 8, 2026 – with Claude

   CollaboratorAuthor

   More actions

   Claim: PM loop round 5 (devx seat)
   Session: session_01F8q5J1MQyocgtNspb15fSn
   Branch: claude/issue-6586-dev-termination-contract
   Worktree: objectstack-issue-6586
   Domain: domain:devx
   File surface: .claude/agents/os-dev.md primarily; .claude/skills/pm-dispatch/SKILL.md only if unavoidable — ⚠️ PR #6720 is in the merge queue editing that file right now, so prefer landing entirely in the agent definition and declare it if not.
   Container: S — mode:subagent, shared container.
   Serial constraints cleared: .claude/agents/os-dev.md last touched by #6691 (merged); no open in-lane claim on it.

   Evidence base has grown twice since filing and both additions are on this thread: 7/7 today including three dispatches that carried an explicit termination-contract clause and died anyway (so the clause is measurably not sufficient), and a fifth distinct sub-shape — an agent that terminated correctly while its five background monitors each fired a full completion notification, one of them watching a process the agent had itself killed.

   ⚠️ The dispatch will state plainly what this card can and cannot fix: the 3/4 clause-carrying failure rate points at something infrastructural (the process ending between PR push and report turn), which a documentation change cannot repair. What it can do is make the PM-side backstop explicit and make a stale monitor identifiable without re-reading the deliverable. Every death so far was fully recoverable from transcript with zero work lost — the cost is latency, not correctness, and the fix should say so rather than implying it eliminates the failure.

   ---

   Generated by Claude Code
7. os-project-manager commented on Aug 8, 2026

   

   os-project-manager

   on Aug 8, 2026 – with Claude

   CollaboratorAuthor

   More actions

   ACCEPT — PR #6740 (devx seat review, round 5).

   One file (.claude/agents/os-dev.md), +67/−2: a new "Terminating cleanly" section after Definition of done, plus two existing sentences repaired.

   Review notes:

   • The honesty clause is present and load-bearing, which was the whole point of the dispatch. The text states that following it does not mean the dev will be heard — 7/7 on 2026-08-08, three of which carried this clause verbatim and failed anyway — and draws the two binding consequences: never read your own silence as success, and the PM's probe-and-revive loop is the standing backstop, not an exception path. It also says the cost is latency and not correctness specifically so a revived dev does not "recover" by redoing the work or opening a second PR — a failure mode I had not thought to name.
   • The monitor rule is stated as a binding, not a habit: a monitor is bound to its own deadline, never to its subject's lifetime; cancel a watched process, cancel its monitor in the same step. Anchored to the measured case (publish/lint 面对字面量空组合子响亮拒收 —— #5322 拍板单位元后的可选编写期信号(PD #12 形状) #5330 / PR feat(lint): refuse literal empty combinators at authoring time (#5330) #6703, six notifications, one from a monitor watching a run the agent had itself TaskStopped).
   • Resource rule 6 was contradicting the new evidence and got fixed in the same pass. It said a background watcher's wake-up "never arrives" — flatly false against publish/lint 面对字面量空组合子响亮拒收 —— #5322 拍板单位元后的可选编写期信号(PD #12 形状) #5330, where five of them arrived. The new text reconciles both: a monitor fires on its own deadline, not on the dev's need, so it will neither rescue a mid-task stop nor stay quiet after the work is done. Same missing binding seen from two sides. Catching that the addition falsified a neighbouring rule is the part most people skip.

   ⚠️ The dev declined the issue's direction 2 and argued it is inverted — I agree, and it bears directly on the open #6644 L2 decision. The card proposed moving the remote-CI read to the PM. But pm-dispatch/SKILL.md forbids the PM from subscribing to a dev's PR before the report lands ("报告前是 dev 的领地,双驾驶员互踩"), and the agent definition's CI-convergence bullet exists because #5584 merged red and poisoned main's merge ref. So the PR keeps the CI wait and instead makes explicit that it is a foreground read ending in the report, never a licence for a background waker. The ownership sentence records the handoff point rather than moving it. I have cross-posted this to #6644 — it is a substantive argument against that card's option B that the maintainer should weigh.

   • SKILL.md correctly untouched (PR docs(pm-dispatch): sweep of 7 measured SKILL corrections — os-regen list drift, the unlock sweep's three duties, verification blindness, the auto-merge signature #6720 was in the merge queue on it); the PM-side half is left as an explicit follow-up on top of it, with the pointer recorded.
   • "No package code changed, so there is no meaningful pnpm test for this diff — saying so beats manufacturing evidence that fits the template." Correct, and the gate list was still run in full, including check:skill-frame-sync, the one that reads this exact file.

   Ready + auto-merge armed 13:52Z.

   ---

   Generated by Claude Code
8. os-project-manager

   closed this as completedin #6740on Aug 8, 2026
9. os-project-manager commented on Aug 9, 2026

   

   os-project-manager

   on Aug 9, 2026 – with Claude

   CollaboratorAuthor

   More actions

   A measurement that extends this card's failure mode — and that contradicts the recovery half of the standing backstop. Recorded by the devx PM seat (session_01F8q5J1MQyocgtNspb15fSn), unassigned, for triage.

   What was measured

   Four os-dev agents dispatched ~23:30Z (issues #6801 / #6803 / #6750 / #6745, PRs #6834 / #6836 / #6835 / #6839, all model: "opus" explicitly). All four opened correct draft PRs with real commits, then went silent — the documented shape of this card.

   Two probe rounds followed, per the pm-dispatch SKILL's probe-and-revive clause:

   round	time	result
   1	00:43Z	4/4 returned had no active task; resumed from transcript in the background
   2	01:10Z	4/4 returned the same thing again

   Between the two rounds the revived agents ran for ~27 minutes and produced nothing observable: no push on any of the four branches (last commits 23:38Z / 23:40Z / 23:42Z / 23:57Z, unchanged), no draft→ready flip, no report delivered, and no completion notification.

   Why this is not just another instance

   The SKILL's clause says the cost of a lost report is latency, not correctness, because "至今每一例都能从 transcript 完整复活,零工作丢失". That is what makes probing the correct and cheap response, and re-dispatch forbidden.

   This batch breaks the second half of that claim. The work already on the branches is indeed intact — nothing was lost. But revival did not recover the report, and the revived run died again leaving no trace it had run at all. Eight probes across two rounds produced zero reports. So on this signature:

   • probing is still safe and still the right first response;
   • but it is no longer reliably recovering — treating "probe again" as the standing answer costs a full patrol cycle per round with nothing to show;
   • and ListAgents is not a usable liveness read either: it reported No reachable agents at 00:42Z while those same four agents had demonstrably done real work. The only authoritative liveness signal is the SendMessage return string.

   What the PM did instead

   Stopped probing after round 2 and moved to the SKILL's own fallback ("报告丢失 ≠ 验收停摆"), whose second condition — probe-confirmed dead — is satisfied twice over. The four PRs will be adjudicated from their artifacts (body, diff, CI) rather than from reports that are not coming.

   Suggested addition to the clause, for whoever picks this up

   The clause currently has one recovery path and treats it as always working. It needs a bounded one: probe at most twice; if the second probe also returns no active task AND the branch has not moved between the two rounds, stop probing and adjudicate from the artifacts. Without a bound, a PM that follows the clause literally re-probes forever, and the "cost is only latency" reasoning quietly stops being true.

   ---

   Generated by Claude Code
10. os-project-manager commented on Aug 9, 2026

    

    os-project-manager

    on Aug 9, 2026 – with Claude

    CollaboratorAuthor

    More actions

    Correction to my previous comment — its central claim was wrong, and the corrected version says something more useful.

    Minutes after I posted it, all four reports arrived, delivered by the round-2 probe. So this line from that comment is false and should not be carried forward:

    revival did not recover the report, and the revived run died again leaving no trace it had run at all

    What actually happened: every one of the four agents had finished its work and emitted its report — twice — before I ever probed. Three of them said so unprompted, in near-identical words: "I have emitted this report twice already and it evidently isn't reaching you." Round 1's probe did not "fail"; its report was lost in the same lossy handback this card is about. Round 2 got through.

    Corrected reading of the measurement:

    claim in the previous comment	status
    4/4 probes returned had no active task in both rounds	stands
    no branch movement between rounds	stands — and it was never evidence of failure: the work was already complete at 23:38–23:57Z, so there was nothing left to push
    "revival did not recover the report"	false — round 2 recovered 4/4
    "the revived run died again leaving no trace it had run at all"	false — it re-emitted the report; the emission did not reach the PM
    ListAgents is not a usable liveness read	stands, and is worth keeping: it reported No reachable agents while those agents had done real work

    The lesson that survives, restated. The failure is entirely in the handback channel, exactly as this card says; the recovery path works but is not reliably one round. A PM that probes once, sees nothing, and concludes the agent is unrecoverable will discard four completed deliverables — which is the mistake I was one step away from making. So the bound I proposed should be inverted in spirit: not "stop probing after two rounds", but "a silent branch is not evidence of an incomplete task — re-probe before you ever consider adjudicating without a report, and read no push since T as possibly finished, not possibly dead."

    Everything downstream held: all four PRs are complete, CI-green, and were accepted on their merits. Zero work lost — which is this card's original claim, and it survived intact.

    ---

    Generated by Claude Code