Skip to content

Name transport failures and local cancellation in worker logs and request metrics #167

Open

Description

@rmcdaniel

Python worker logs and request metrics do not let operators tell an intermediary disconnect or client timeout apart from a server refusal or a local cancellation.

Current behavior

Main 150c6f6bc14e3516f75cc615b2f75f093ed84ea1:

  • _poll_error_summary returns str(error) for anything that is not a ServerError. Poll warnings log only that summary, with no exception class. httpx timeout and read errors can have an empty message, so the warning can name nothing.
  • The request metric starts with outcome = "pending" and updates it only for Exception. asyncio.CancelledError is not an Exception, so a request cancelled by shutdown or by the caller is recorded with outcome="pending" and status_code="none".
  • The Python docs do not list the outcome values or say that transport failures surface as raw httpx exceptions after retries run out.

This was found by reading source and the pinned dependency code. It was not reproduced against a live proxy.

Expected

Poll and request failure logs always name the exception class and route. Local cancellation has an explicit documented outcome. The docs separate transport failures, durable activity or workflow cancellation, and asyncio.CancelledError.

Acceptance

  • Unit tests: a message-less ReadTimeout or ReadError produces a warning that names the class, and server refusals keep their status and reason.
  • Cancelling an in-flight request records the documented cancellation outcome.
  • Document the metric outcome values and the error categories.
  • Decide whether to retry RemoteProtocolError for idempotent worker requests, document the decision and test it.
  • Link from durable-workflow.github.io#267.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    priority:P3Lower-priority product improvement
    status:readyReady for implementation
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingkind:defectA public product behavior is incorrectpriority:P3Lower-priority product improvementstatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions