Skip to content

Sweep leftover path identity probes in startup recovery (#181) - #192

Merged
mchwang merged 6 commits into
mainfrom
fix/181-probe-sweep
Oct 21, 2026
Merged

mchwang merged 6 commits into
mainfrom
fix/181-probe-sweep

Conversation

@mchwang

@mchwang mchwang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #181

What changes

  • A label shape for runner-scoped objects (agents/labels.ts). PROBE_LABEL is io.codeboost.probe, and runnerProbeLabels(runnerOwner, 'path-identity') validates the token. A probe carries the runner label and its kind, and no attempt or allocation label.
  • The path identity probe is labelled (agents/container/storage.ts). pathIdentityProbeArgs puts both labels on the container, and with volume-label= on its anonymous volume as well. probeTaskStoragePathIdentity(imageId, runnerOwner, …) takes the database's runner token, which setUpRunner passes.
  • Startup recovery sweeps leftover probes (agents/recovery.ts), under the runner lock and before this process probes:
    • It removes this runner's probe containers by full ID, then their volumes by name. No volume or network is attempted after a failed removal.
    • A probe container is removed only if it has the probe label with value path-identity, no attempt, allocation, invocation, egress or storage label, and a name codeboost-path-identity-<uuid v4>.
    • A probe volume Docker reports "in use" is kept, not treated as a failure. Its holder is a container recovery leaves, and a start after that holder is gone removes the volume.
    • An object listed under this runner but inspected with another runner's label replaced the listed one. Recovery neither removes it nor reports it.
    • A probe volume is removed only if it has the same labels and a 64-hex anonymous name.
    • Any other combination is reported as unknown-kind. A probe without a runner label is reported as no-runner-label; PROBE_LABEL was added to the unowned scan for this.
  • The maintainer's note on the volume: the volume is now labelled, so a volume that --rm or rm --volumes failed to remove is found and removed by the next startup's recovery.

Tests

  • Real Docker (test/runner-workspace.test.ts): a probe left by a killed runner, created from the probe's own arguments, is removed with its volume. Another runner's probe stays, and so does this runner's container without the probe kind label.
  • Fake daemon (test/agent-recovery.test.ts, answering volume rm/volume inspect as Docker 27.4.0 does):
    • the container is removed first, then its volume, and no other runner's probe is touched;
    • five label and name combinations are left and reported;
    • a probe without a runner label is reported;
    • a refused container blocks its volume's removal;
    • a refused volume removal fails recovery, and no network is removed after it;
    • a lone volume whose container is gone is removed;
    • a volume a container still mounts is kept;
    • an object relabelled between the list and the inspect is neither removed nor reported.
  • Fake docker (test/path-identity-probe.test.ts): the exact labelled arguments, and a malformed token refused before any Docker call.
  • test/runner-production.test.ts: the probe gets the token recovery ran under.

Review record

Each independent round was a fresh general-purpose agent with no shared context, reviewing the full diff against origin/main. Base aced300 for rounds 1–4; base 46567ea (after #190 merged) for round 5.

Round Reviewer Head Findings What changed
Self author 7bf5369 — —
1 independent agent 7bf5369 Minor: docs said "exactly those labels"; the runner-lifecycle row was missing the probe exception; the comment "none is still in use" was untrue with an orphaned detached client; the fake daemon's volume answers were not Docker's. Nits: no test for the container-first stop or for a malformed token 044a7da
2 independent agent 044a7da Minor: an in-use probe volume would fail every start. Nits: the runner label was not rechecked for volumes removed by name; doc wording; no lone-volume test f422878 (in-use volume kept, runner-label check, tests)
3 independent agent f422878 Minor: a kept volume was silent, so the wording was reworded; the fake rm --force of a missing container was not Docker 27.4's. Nits: comment scope; a relabelled volume was reported as this runner's e1b5b32 (relabelled objects dropped after the inspect; fake fixed)
4 independent agent e1b5b32 Minor: two comments placed the orphaned-client window wrongly. Nits: the fixture dropped --rm; the failure message did not mention recovery 266564e
— merge of main (#190) 94b21ae — —
5 independent agent 94b21ae CLEAN. It checked every fake answer against Docker 27.4.0, and traced each new test against broken logic —
Copilot Copilot 94b21ae Requested after the clean round 5. No open issues. Its summary-only concern was that real-Docker validation was unconfirmed Answered under "Validation at the reviewed head" below. No code change

Validation at the reviewed head (94b21ae, base 46567ea):

  • real-docker passed in 21m6s (run 39530270190). It ran the new real-Docker recovery test in test/runner-workspace.test.ts.
  • test passed in 24m15s (run 39530270067).
  • Locally, npm run typecheck passes, and so do test/agent-recovery.test.ts, test/path-identity-probe.test.ts and test/runner-production.test.ts.

Declined (round 5 observations, with the reason):

Review lessons (merge gate):

  • The orphaned-client comments and the version-checked fake answers (rounds 1–4) are covered by CLAUDE.md: "A fake of an external tool … must give … each part of the answer the code reads as the real tool gives it", and "Treat every behavioural claim … verify".
  • The rest were one-off fixes.

🤖 Generated with Claude Code

mchwang and others added 6 commits October 20, 2026 18:05
The probe's container and anonymous volume now carry the database's runner
token and `io.codeboost.probe=path-identity`, a label shape for runner-scoped
objects. Startup recovery, under the runner lock, removes this runner's probe
containers (by full ID) and then their volumes (by name), and leaves anything
of another runner, anything without the probe kind, and anything with attempt,
allocation or other kind labels. A probe without a runner label is reported.

Closes #181

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…181)

Review round 1: the fake daemon exits 0 for a missing volume under --force
and refuses a volume a container mounts; tests for a refused probe container
and a malformed runner token; docs state the probe exception to the scoped
recovery rule and what an orphaned client can do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Review round 2: a probe volume that a container recovery leaves still mounts
is kept for a later start rather than failing every start; a probe volume
counts only with this runner's label, since it is removed by name; a lone
leftover volume is tested; doc wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 27.4 (#181)

Review round 3: an object listed under this runner but inspected with
another's is neither removed nor reported; the fake's rm --force of a missing
container exits 0, as Docker 27.4 does; the kept in-use volume's wording says
it goes once its holder is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e fixture (#181)

Review round 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mchwang
mchwang marked this pull request as ready for review October 21, 2026 02:03
@mchwang
mchwang requested a balanced review from Copilot October 21, 2026 02:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Destructive Docker cleanup needs human confirmation of real-Docker validation, which remains unverified in this review.

No open issues 🎉

What changed in this PR

Addresses #181 by making interrupted path identity probes discoverable and removable during runner startup recovery.

Changes:

  • Labels probe containers and volumes with the runner token and probe kind.
  • Adds ownership-checked cleanup, removing containers before volumes and networks.
  • Adds regression coverage and updates recovery documentation.
File Description
test/​runner-workspace.test.ts Adds real-Docker recovery and runner-isolation coverage.
test/​runner-production.test.ts Verifies recovery and probing use the same runner token.
test/​path-identity-probe.test.ts Checks labelled arguments and invalid-token rejection.
test/​agent-recovery.test.ts Covers cleanup ordering, ownership checks, and failures.
runner/​production.ts Passes the runner token to the probe.
docs/​implementation/​runner-lifecycle.md Documents startup probe recovery.
docs/​implementation/​agent-isolation.md Describes probe ownership and cleanup rules.
docs/​architecture.md Records the probe-recovery follow-up.
agents/​recovery.ts Recognizes and removes owned leftover probes.
agents/​labels.ts Defines validated runner-scoped probe labels.
agents/​container/​storage.ts Labels probe containers and anonymous volumes.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sweep leftover path identity probe containers in startup recovery

2 participants