Repository navigation
fix(worker): a terminal write to a dropped schema drops the task instead of the worker - #5813
Merged
Merged
Conversation
…ead of the worker (#5811) Deleting a tenant drops its schema, async_operations included. A task still in flight then fails every terminal status write with "relation does not exist". _write_terminal treated that like an unreachable database: it retried for TERMINAL_WRITE_DEADLINE_S (300s) and then SIGTERMed the worker, taking every other in-flight task on that process down with it, so that restart recovery could hand back a row that no longer exists. In a combined API+worker process the API goes down too, so a suite that deletes organizations while async work is in flight fails partway through with ECONNREFUSED. A failed terminal write is now checked before it is retried: if the error says a table or schema does not exist (Postgres 42P01/3F000, Oracle ORA-00942) AND a read of the task's own operations table fails the same way, the row is gone with its schema, so the write is abandoned with a warning and the worker keeps running. A missing table elsewhere in the write, or a probe that fails for any other reason, keeps the existing retry-then-exit path, which still protects real DB outages (#3228, #5377). Tests: a dropped schema returns without exiting (fails without the fix); a missing table while the operations table exists, and an unreachable DB, both still shut the worker down; the missing-relation check sees through exception wrapping and recognises the Oracle code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5811.
Problem
Deleting a tenant drops its schema,
async_operationsincluded. A task still in flight then fails every terminal status write with "relation does not exist"._write_terminaltreated that the same as an unreachable database:TERMINAL_WRITE_DEADLINE_S(300 s);In a combined API + worker process the API goes down too. A suite that deletes organizations while async work is in flight then fails partway through with
ECONNREFUSED.Fix
Before retrying, a failed terminal write is now checked. If both of these hold:
42P01/3F000, OracleORA-00942, found anywhere in the exception chain), andthen the row is gone with its schema. The write is abandoned with a warning, and the worker keeps running.
The existing retry-then-exit path is kept in two cases, so real DB outages are still covered (#3228, #5377):
Tests
tests/test_worker.py::TestTerminalWriteToADroppedSchema:tests/test_worker.py: 146 passed.ruff checkandruff formatare clean on the changed files.