Repository navigation
Remove test objects made without a runner label when the test fails early #189
Description
Problem
Some tests in test/agent-container.test.ts deliberately create Docker objects without the io.codeboost.runner label, to prove that the owner-label checks reject them. Each test removes its own objects as the last step of its body. If the test fails or times out before that step, the objects stay on the daemon, and no cleanup can find them:
afterAll'sremoveRunnerObjects(TEST_RUNNER_OWNER)(Keep each Docker test run's containers and counts to itself #185) filters by the runner label, which these objects lack.refuseThenRemovetakes the allocation out oftaskFilesystemsfirst, so the ordinaryafterAllcleanup also skips it.- The production cleanup refuses an object without every owner label, by design.
Evidence
A keeper container ran on this machine's daemon for 6 days. Its labels were io.codeboost.task-storage=keeper and io.codeboost.attempt=fixture, plus an allocation ID and no runner label. Attempt fixture is the default from testOwner().
That matches "rejects a keeper without its runner label before the agent starts" (test/agent-container.test.ts, about line 672) stopping before refuseThenRemove:
const data = await withoutRunnerLabel([['create', '--name']], () => fixture());
const live = await profile(data, 'planning', 'must-not-run');
await expect(createValidatedContainer(live)).rejects.toThrow('trusted keeper'); // a failure or timeout here…
refuseThenRemove(data.filesystems, [['rm', '--force', data.filesystems.keeper]]); // …skips the removalBy 2026-10-20 the container was gone; someone removed it by hand.
Affected tests
These tests leave objects without a runner label if they stop early:
- agent-container: "rejects task volumes without their runner label before the agent starts" leaves the work and metadata volumes.
- agent-container: "rejects a keeper without its runner label before the agent starts" leaves the keeper. The storage it made also stays: its volumes carry the runner label, but the keeper still holds them in use.
- agent-container: "rejects an agent container without its runner label before it starts": check this one. The test asserts the container is gone, but does not remove it if the assertion fails.
- agent-network: the impostor, blocker and peer containers are removed in
finallyblocks, so these look safe. Confirm.
Suggested fix
Pick one approach and apply it to every test that creates objects without a runner label:
- Move the removal into
onTestFinishedortry/finally. Register the label-less objects' removal as soon as they exist, so the removal runs however the test ends. - Add a run label the shim does not strip. For example, add
io.codeboost.test-run=<per-run id>to every object the test creates, and have theafterAllsweep also remove objects by that label.
Done when
- A test that creates an object without a runner label removes it even when it fails or times out after creating it.
- A regression check proves that, for example, by forcing the assertion before
refuseThenRemoveto fail and then confirming no object with that allocation ID remains.
Related
- Keep each Docker test run's containers and counts to itself #185 and Give the network, gate and workspace Docker tests per-run isolation #188 added per-run owners and the owner sweep.
- Sweep leftover path identity probe containers in startup recovery #181 covers the same leak class for the path identity probe in production.
- Separately, this daemon has four exited keepers, 13 days old, with random runner labels. Those came from runs before the sweep existed and are not this bug.
Activity
Remove only exact holders of a swept network; test the holder sweep (#…
Close the remaining label-less leaks the review found (#189)
Record label-less objects first in agent-network; remove networks by …
Test removal by full network ID; say what the runner sweep removes (#189
Remove swept containers by full ID; track the blocker before it is ma…
Remove the foreign-runner storage in its test; clean every storage in…
Read network names as JSON; name the seccomp probe; test --no-trunc (#…
Never throw on output the sweep cannot parse; foreign creates inside …
Sweep the failed-allocation test by allocation; guard unreadable lines (
Budget the in-test sweeps; exact wording (#189)
Sweep both runners of the recovery test; keep the first failure (#189)
Check the traps are live in the holder test; report every cleanup fai…
Keep the first failure in the replaced-keeper test; exact sweep wordi…
Check the partial-inspect answer; sweep the replaced keeper; budget m…
Budget the replaced-keeper sweep; say what the holder test's budget c…
43 remaining items
Report sweep lines of the wrong shape; exact commands in removeByName…
Gone checks need Docker's missing answer; sweep checks listed IDs and…
agent-network's survival checks use objectExists (#189)
Every ID passed on as an argument has a checked shape (#189)
removeByName refuses a name that could read as an option; fake notes (#…
Test objectExists; gone cases reach their inspect; per-record errors …
Gone checks in the touched tests use objectExists; fake and comment f…
Every gone check in a changed test uses objectExists (#189)
The proxy-start test's gone check uses objectExists; assert the raced…
Shape checks before the gone checks; objectExists refuses an option-l…
Trim the listing's stderr in its failure message (#189)
Pin every docker call's time limit; volume refusal reason; guarded re…
Time-limit test runs all thirteen calls; holder search and no-space l…
Test the fixture's remaining guards; exact reasons in refusals and re…