Skip to content

fix(worker): a terminal write to a dropped schema drops the task instead of the worker - #5813

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/terminal-write-dropped-schema
Oct 20, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/terminal-write-dropped-schema

Conversation

@cdbartholomew

Copy link
Copy Markdown
Contributor

Fixes #5811.

Problem

Deleting a tenant drops its schema, async_operations included. A task still in flight then fails every terminal status write with "relation does not exist". _write_terminal treated that the same as an unreachable database:

  • it retried for TERMINAL_WRITE_DEADLINE_S (300 s);
  • then it SIGTERMed the worker, taking every other in-flight task on that process down with it;
  • the purpose was to let restart recovery hand back the row, but that row no longer exists.

In a combined API + worker process the API goes down too. A suite that deletes organizations while async work is in flight then fails partway through with ECONNREFUSED.

Fix

Before retrying, a failed terminal write is now checked. If both of these hold:

  1. the error says a table or schema does not exist (Postgres 42P01/3F000, Oracle ORA-00942, found anywhere in the exception chain), and
  2. a read of the task's own operations table fails the same way,

then the row is gone with its schema. The write is abandoned with a warning, and the worker keeps running.

The existing retry-then-exit path is kept in two cases, so real DB outages are still covered (#3228, #5377):

  • A missing table elsewhere in the write. The row is still there and could be a real bug to surface.
  • A probe that fails for any other reason. That isn't proof the schema is gone.

Tests

tests/test_worker.py::TestTerminalWriteToADroppedSchema:

  • Dropped schema: returns without exiting. This test fails without the fix, because the worker exits.
  • Missing table while the operations table exists: still shuts the worker down.
  • Unreachable DB: still shuts the worker down.
  • The missing-relation check: sees through exception wrapping and recognises the Oracle code.

tests/test_worker.py: 146 passed. ruff check and ruff format are clean on the changed files.

…ead of the worker (#5811)

Deleting a tenant drops its schema, async_operations included. A task still
in flight then fails every terminal status write with "relation does not
exist". _write_terminal treated that like an unreachable database: it
retried for TERMINAL_WRITE_DEADLINE_S (300s) and then SIGTERMed the worker,
taking every other in-flight task on that process down with it, so that
restart recovery could hand back a row that no longer exists. In a combined
API+worker process the API goes down too, so a suite that deletes
organizations while async work is in flight fails partway through with
ECONNREFUSED.

A failed terminal write is now checked before it is retried: if the error
says a table or schema does not exist (Postgres 42P01/3F000, Oracle
ORA-00942) AND a read of the task's own operations table fails the same
way, the row is gone with its schema, so the write is abandoned with a
warning and the worker keeps running. A missing table elsewhere in the
write, or a probe that fails for any other reason, keeps the existing
retry-then-exit path, which still protects real DB outages (#3228, #5377).

Tests: a dropped schema returns without exiting (fails without the fix);
a missing table while the operations table exists, and an unreachable DB,
both still shut the worker down; the missing-relation check sees through
exception wrapping and recognises the Oracle code.
@nicoloboschi
nicoloboschi merged commit 453b2a5 into main Oct 20, 2026
117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/terminal-write-dropped-schema branch October 20, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker shuts itself down when a terminal status write fails because the tenant schema was dropped

2 participants