Repository navigation
fix(jobs): record _job_start_time on the server's clock - #1622
dimitri-yatsenko wants to merge 1 commit into
Conversation
populate() wrote _job_start_time as the populating host's local time, with no zone: time.time() at the start of the job, converted by datetime.fromtimestamp(). Two hosts in different time zones recorded different values for the same instant, and nothing in the row said which zone applied. The value could not be placed beside the job table's reserved_time and completed_time, which come from the server. The start time is now computed in the UPDATE that already writes the hidden columns, as the server's current time less _job_duration, so no query is added. The expression is the start of the statement, not of the transaction: NOW(3) on MySQL, and statement_timestamp() on PostgreSQL, where CURRENT_TIMESTAMP is the start of the transaction. It comes from a new adapter method, statement_timestamp_expr(). _job_duration is still measured on the populating host, now with time.monotonic(), which neither the host's zone nor an adjustment to its clock affects. The UPDATE runs just after the commit that the duration includes, so the derived start is late by the time that one statement takes to reach the server. Rows written before this change keep the populating host's local time. They cannot be corrected, because the zone was never recorded. The test populates two keys as if from hosts in UTC-12 and UTC+14 and checks, on MySQL and PostgreSQL, that each _job_start_time falls between two readings of the server's clock taken around the call. Without the change it fails on both, 12 hours off. Closes #1584.
MilagrosMarin
left a comment
There was a problem hiding this comment.
Thanks @dimitri-yatsenko, nice fix, and computing the start in the UPDATE that already runs avoids an extra round trip.
Verified locally (MySQL + PostgreSQL via testcontainers):
✅ test_hidden_job_metadata.py passes (12 tests)
✅ With master's autopopulate.py restored, test_job_start_time_uses_server_clock fails on both backends, 12 hours off, so the test guards the fix
✅ NOW(3) - INTERVAL n MICROSECOND / CAST(statement_timestamp() AS timestamptz(3)) - INTERVAL 'n microseconds' match the datetime(3) column, and now share a clock with reserved_time / completed_time
One note, not blocking: on PostgreSQL the timestamptz → timestamp assignment uses the session TimeZone, which a client can override (e.g. PGTZ); MySQL's session time_zone behaves the same way. reserved_time/completed_time go through the same conversion, so the three stay consistent, but "one clock for every host" holds as long as clients keep the server's default session zone. A line in the _update_job_metadata docstring might be worth it.
populate() wrote _job_start_time as the populating host's local time, with no zone: time.time() at the start of the job, converted by datetime.fromtimestamp(). Two hosts in different time zones recorded different values for the same instant, and nothing in the row said which zone applied. The value could not be placed beside the job table's reserved_time and completed_time, which come from the server.
The start time is now computed in the UPDATE that already writes the hidden columns, as the server's current time less _job_duration, so no query is added. The expression is the start of the statement, not of the transaction: NOW(3) on MySQL, and statement_timestamp() on PostgreSQL, where CURRENT_TIMESTAMP is the start of the transaction. It comes from a new adapter method, statement_timestamp_expr().
_job_duration is still measured on the populating host, now with time.monotonic(), which neither the host's zone nor an adjustment to its clock affects. The UPDATE runs just after the commit that the duration includes, so the derived start is late by the time that one statement takes to reach the server.
Rows written before this change keep the populating host's local time. They cannot be corrected, because the zone was never recorded.
The test populates two keys as if from hosts in UTC-12 and UTC+14 and checks, on MySQL and PostgreSQL, that each _job_start_time falls between two readings of the server's clock taken around the call. Without the change it fails on both, 12 hours off.
Closes #1584.