Skip to content

F2b: durable safety findings, failed-run audit and partial output (#87, part 2) - #94

Merged
mchwang merged 3 commits into
mainfrom
feat/f2b-findings-export
Oct 1, 2026
Merged

mchwang merged 3 commits into
mainfrom
feat/f2b-findings-export

Conversation

@mchwang

@mchwang mchwang commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Part 2 of #87: items 1–4. It also closes the through-link gap left from part 1. Part 1 merged as #92; production wiring is #91.

What it adds

Item What it does
3. Durable safety findings A finding is saved on its attempt (schema v8, attempts.safety_finding) through the shutdown capability, before the terminal write. The terminal write then sends the task to needs human in the same transaction, whatever outcome wins: a stop or a stale context does not undo what the agent did. A pending cancel still closes the task. Startup recovery does the same for an attempt a crash left, and does not requeue it. The migration is idempotent.
4. Findings outside ItemExecutor The Store acts on the finding, so an attempt started through /api/runner start or retry goes to needs human too, and admission refuses new work from needs human.
2. Auditing failed runs New RunnerDeps.auditFailed. When the agent ends badly on its own (a non-zero exit or a D stop reason, with no first reason), the same inspect-and-audit as finish runs before the terminal write. A violation, or an inspection that refuses, is a finding. Nothing is committed. A stopped, stale, time-limited or shut-down run is not audited.
1. Partial output New RunnerDeps.exportPartial and TaskWorkspace.exportPartial (D's exportTaskDiff). For a writable attempt that did not complete, the diff is saved as <diagnostics>/<attemptId>.diff, owner-only, through an exclusive temporary file, and stored as diagnostic_ref by the terminal write. The export has its own 60 s deadline and is never cut short by a stop. A failure adds a quoted "Partial output could not be exported" to the diagnostic.
Retention (runner/diagnostics.ts) Over the cap (256 MiB by default): unreferenced files go first, oldest first. Then the oldest referenced file: its reference is cleared and noted in one transaction (forgetDiagnostic), and only then is the file deleted.
through-link (from part 1) A declared link whose snapshot status is through-link is never launched. Preparation records a finding and fails.

Design note

A task cannot change status while it has an active attempt (transitionTask refuses). So the terminal write always meets a running task, and a saved finding is always acted on there. I first built an "owed" mechanism (admission and merge refusals, a pay step), then removed it as unreachable. Only a finding whose save failed is still owed, in memory (SafetyFindings), exactly as before.

Test changes

  • Fourteen executor tests covered the executor's own handling of a finding. Twelve now take that path through a new findingSaveError option, keeping their meaning. Two described behaviour this changes and assert the new behaviour:
    • the finding is acted on at the terminal write, before release can set a gate;
    • an attempt started outside the executor goes to needs human and nothing more is admitted.
  • New tests:
    • durable findings in the Store, recovery and the v7 to v8 migration;
    • failed-run audits;
    • partial output and its failure note;
    • retention ordering;
    • the through-link refusal;
    • a real-Docker check that a scope pause stays approvable after a review load.

Validation (local; first pushed head e07f478, later heads in the rounds below)

  • npm run typecheck: passes.
  • Regular test job: 1186 of 1187. The one failure is the known load timeout in review.test.ts, which passes alone (15 of 15) at a load average of about 70.
  • test/runner-workspace.test.ts (real Docker): 3 of 3.

Review rounds (/code-review high)

Round Fixed Declined
1 • A result that is not this attempt's skipped the failed-run audit and the partial-output export; both now run in one place (#keepEvidence).
• The "stopped run" test stopped before launch; it now stops during the run.
• The export deadline is tested.
forgetDiagnostic raising the owning task's state version: the Store's rule is that every durable change to an attempt raises it, and a "stale state" refusal is fail-safe (the person reloads).
2 A clean run whose finish failed was inspected a second time; now only a run that ended badly on its own is audited. None
3 Nothing new. None

The new tests for rounds 1 and 2 were checked to fail with their fix removed.

Review lessons (merge gate)

  • Foreign result skipping evidence: one-off. Fixed by funnelling both paths through one helper.
  • Test that never reached the branch it named: covered by "Review readiness": "A test's setup must leave the state the production path would".
  • Double audit: one-off. A refactor widened a condition; covered by a test that counts inspections.

🤖 Generated with Claude Code

mchwang and others added 3 commits October 1, 2026 11:48
… part 2)

- Safety findings are saved on their attempt (schema v8, safety_finding)
  before the terminal write, which sends the task to needs human in the
  same transaction whatever outcome wins; startup recovery does the same
  and does not requeue. An attempt started outside ItemExecutor is covered
  too. Only a finding whose save failed is held in memory, as before.
- A run whose agent failed on its own is inspected and audited
  (auditFailed); a violation or a refusing inspection is a finding. Nothing
  is committed.
- A declared link whose snapshot status is through-link is never launched.
- A writable attempt that did not complete keeps its partial output
  (exportPartial): D's bounded diff saved owner-only and referenced as
  diagnostic_ref, with retention that never deletes a referenced file.
- Test: a scope pause stays approvable after a review load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A result that is not this attempt's no longer skips the failed-run audit
  and the partial-output export: both run in one place (#keepEvidence).
- Tests: a run stopped while it ran is not audited (the old test stopped
  before launch), and an export gives up at its own deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w round 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mchwang
mchwang merged commit dfda871 into main Oct 1, 2026
2 checks passed
@mchwang
mchwang deleted the feat/f2b-findings-export branch October 20, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant