Skip to content

Sweep leftover path identity probe containers in startup recovery #181

Closed

Description

@mchwang

Follow-up from the review of #180.

Problem

At startup the runner probes task storage's path identity with probeTaskStoragePathIdentity (agents/container/storage.ts). This runs a docker run --rm container named codeboost-path-identity-<uuid> on an anonymous tmpfs volume. When the probe fails, it removes that container by name and confirms the removal with container inspect through a 10 s settle window.

Two cases can still leave a created, never-started container and its volume:

  • a create that reaches the daemon after the settle window;
  • a runner that dies (SIGKILL, a second Ctrl+C) after the client is killed but before the removal.

The probe carries no labels, so startup recovery (agents/recovery.ts, which lists by label) never finds it. The container stays until a person removes it with docker ps -a --filter name=codeboost-path-identity-. It runs no process and its unmounted tmpfs volume holds no memory, so the cost is clutter, not resources.

Proposal

Label the probe with the database's runner owner token. Recovery then sweeps containers with that token and the probe kind, under the runner lock. Only one runner holds a given token at a time, so the sweep cannot remove another runner's probe while it runs.

Today's owner labels (agents/labels.ts) require attempt and allocation IDs, which the probe does not have. The work therefore includes a label shape for runner-scoped objects, plus the matching recovery filter and ownership check.

Acceptance

  • A probe container left by a killed runner is removed by the next startup's recovery, together with its anonymous volume.
  • The sweep removes nothing labelled with another runner token, and nothing without the probe kind label.
  • Tests for both, against real Docker in the Docker-backed suite.

Activity

  1. mchwang commented on Oct 20, 2026

    @mchwang
    ContributorAuthor

    Also in scope, accepted by the maintainer when merging #180: the probe's anonymous volume removal is not confirmed. docker run --rm and docker rm --force --volumes remove it, but the daemon only logs a failure to remove a volume, and the probe's cleanup confirms only that the container is gone (container inspect). A failed volume removal leaves an unlabelled 1 MiB tmpfs volume that nothing finds. A fix could give the probe a named, labelled volume, remove it explicitly on every path, and confirm with volume inspect; the recovery sweep here would then also remove leftover probe volumes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions