Skip to content

fix(jobs): stale_timeout=0 means no grace period - #1623

Draft
dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/stale-timeout-zero
Draft

dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/stale-timeout-zero

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

refresh() removes a job whose key has left key_source once the job is older than stale_timeout, a grace period for a key_source that is still being filled. stale_timeout=0 skipped that check instead, so there was no way to ask for the cleanup now: refresh(stale_timeout=0) left every such job in place, and so did populate() with config.jobs.stale_timeout = 0.

0 now means no grace period: on that refresh, every job whose key is no longer in key_source is removed, whatever its age, unless it is marked ignore. That includes reserved jobs, and success records of rows that remain after their key left key_source, which a nonzero stale_timeout keeps for that many seconds. stale_timeout=None still means the configured value (config.jobs.stale_timeout, 3600 seconds by default), so a refresh that does not pass it keeps the grace period. The setting's description, "Seconds before pending job is checked for staleness", now describes the grace period and what 0 does.

Skipping the check is no longer expressible as 0; a long stale_timeout keeps such jobs for as long as it says.

The tests delete upstream rows and check, on MySQL and PostgreSQL, that a default refresh keeps the pending jobs of the deleted keys, that refresh(stale_timeout=0) removes them, and that populate() with config.jobs.stale_timeout = 0 removes them instead of trying to compute them. Each fails without the change. They replace the test that held 0 to its old meaning.

Closes #1577.

Stacked on #1595: its base is fix/jobs-refresh-restriction until that merges.

@dimitri-yatsenko dimitri-yatsenko added this to the v2.4 milestone Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko added the bug Indicates an unexpected problem or unintended behavior label Oct 17, 2026
refresh() removes a job whose key has left key_source once the job is
older than stale_timeout, a grace period for a key_source that is still
being filled. stale_timeout=0 skipped that check instead, so there was no
way to ask for the cleanup now: refresh(stale_timeout=0) left every such
job in place, and so did populate() with config.jobs.stale_timeout = 0.

0 now means no grace period: on that refresh, every job whose key is no
longer in key_source is removed, whatever its age, unless it is marked
ignore. That includes reserved jobs, and success records of rows that
remain after their key left key_source, which a nonzero stale_timeout
keeps for that many seconds. stale_timeout=None still means the
configured value (config.jobs.stale_timeout, 3600 seconds by default), so
a refresh that does not pass it keeps the grace period. The setting's
description, "Seconds before pending job is checked for staleness", now
describes the grace period and what 0 does.

Skipping the check is no longer expressible as 0; a long stale_timeout
keeps such jobs for as long as it says.

The tests delete upstream rows and check, on MySQL and PostgreSQL, that a
default refresh keeps the pending jobs of the deleted keys, that
refresh(stale_timeout=0) removes them, and that populate() with
config.jobs.stale_timeout = 0 removes them instead of trying to compute
them. Each fails without the change. They replace the test that held 0 to
its old meaning.

Closes #1577.

@MilagrosMarin MilagrosMarin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Making 0 mean "no grace period" is the more natural reading, and the change is minimal: the age filter applies only when stale_timeout > 0, and the delete always runs. Both tests would fail under the old behavior: the first shows the default keeps vanished jobs and 0 removes them, and the second shows populate() with config.jobs.stale_timeout = 0 removes job 1 instead of leaving it pending.

Two non-blocking notes:

  1. This changes behavior for anyone who set config.jobs.stale_timeout = 0 to keep jobs whose key has left key_source: after upgrading, the next populate() removes them, reserved ones included. Worth calling out in the release notes, with "set a large stale_timeout" as the way to get the old behavior back.
  2. With 0, a job reserved by a worker that is still in make() is removed too, so its complete()/error() lands after its row is gone. The key has left key_source, so the work is likely moot, but a sentence in the stale_timeout docstring would make that explicit.

(The "Stacked on #1595" line in the description is out of date: #1595 has merged and the base is master.)

@dimitri-yatsenko
dimitri-yatsenko marked this pull request as draft October 21, 2026 16:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Indicates an unexpected problem or unintended behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants