Skip to content

Remove test objects made without a runner label when the test fails early #189

Closed

Description

@mchwang

Problem

Some tests in test/agent-container.test.ts deliberately create Docker objects without the io.codeboost.runner label, to prove that the owner-label checks reject them. Each test removes its own objects as the last step of its body. If the test fails or times out before that step, the objects stay on the daemon, and no cleanup can find them:

  • afterAll's removeRunnerObjects(TEST_RUNNER_OWNER) (Keep each Docker test run's containers and counts to itself #185) filters by the runner label, which these objects lack.
  • refuseThenRemove takes the allocation out of taskFilesystems first, so the ordinary afterAll cleanup also skips it.
  • The production cleanup refuses an object without every owner label, by design.

Evidence

A keeper container ran on this machine's daemon for 6 days. Its labels were io.codeboost.task-storage=keeper and io.codeboost.attempt=fixture, plus an allocation ID and no runner label. Attempt fixture is the default from testOwner().

That matches "rejects a keeper without its runner label before the agent starts" (test/agent-container.test.ts, about line 672) stopping before refuseThenRemove:

const data = await withoutRunnerLabel([['create', '--name']], () => fixture());
const live = await profile(data, 'planning', 'must-not-run');
await expect(createValidatedContainer(live)).rejects.toThrow('trusted keeper'); // a failure or timeout here…
refuseThenRemove(data.filesystems, [['rm', '--force', data.filesystems.keeper]]); // …skips the removal

By 2026-10-20 the container was gone; someone removed it by hand.

Affected tests

These tests leave objects without a runner label if they stop early:

  • agent-container: "rejects task volumes without their runner label before the agent starts" leaves the work and metadata volumes.
  • agent-container: "rejects a keeper without its runner label before the agent starts" leaves the keeper. The storage it made also stays: its volumes carry the runner label, but the keeper still holds them in use.
  • agent-container: "rejects an agent container without its runner label before it starts": check this one. The test asserts the container is gone, but does not remove it if the assertion fails.
  • agent-network: the impostor, blocker and peer containers are removed in finally blocks, so these look safe. Confirm.

Suggested fix

Pick one approach and apply it to every test that creates objects without a runner label:

  1. Move the removal into onTestFinished or try/finally. Register the label-less objects' removal as soon as they exist, so the removal runs however the test ends.
  2. Add a run label the shim does not strip. For example, add io.codeboost.test-run=<per-run id> to every object the test creates, and have the afterAll sweep also remove objects by that label.

Done when

  • A test that creates an object without a runner label removes it even when it fails or times out after creating it.
  • A regression check proves that, for example, by forcing the assertion before refuseThenRemove to fail and then confirming no object with that allocation ID remains.

Related

Activity

  1. 43 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions