You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
batchTriggerAndWait parent never resumes after all child runs complete (batch stuck in PROCESSING with runs: []) #4971
A task that calls batchTriggerAndWait never resumed after every child run in the batch reached COMPLETED. The parent stayed frozen (no attempt error, durationMs: 0, no output) for ~95 minutes until we cancelled it by hand.
Three parent runs of the same task (scheduled-metrics-orchestration) hit this in one wave on 2026-09-21. Their siblings (17 other runs of the same task, same code, same batch sizes) resumed normally within a minute of their children completing.
Two shapes were observed:
1. Batch stuck in PROCESSING although all its runs finished (2 of 3 cases)
GET /api/v1/batches/{id} still returns status: "PROCESSING" and runs: [] hours after every child run completed:
Parent run
Batch
runCount
Batch created
Last child finishedAt
Batch status now
run_06gccj5uiktpn7mrncb3mki501
batch_06gcclsm9uuel78rsk0jsgoa01
562
2026-09-21T23:40:24Z
2026-09-21T23:41:13Z
PROCESSING, runs: [], updatedAt 23:40:24Z
run_06gccj5ugpgo8odb5kr8incm01
batch_06gcclu8fqilo4lfhncod8b401
76
2026-09-21T23:40:36Z
2026-09-21T23:41:01Z
PROCESSING, runs: [], updatedAt 23:40:37Z
All 562 (resp. 76) child runs show status: COMPLETED under the parent's relatedRuns.children, and each child carries the batch id above. The batch record itself never transitioned out of PROCESSING, so it looks like the batch's run list was never sealed and the parent's waitpoint was never completed.
2. Batch COMPLETED, parent still never resumed (1 of 3 cases)
Parent run
Batch
runCount
Batch updatedAt
Parent resumed?
run_06gccj5uhhi8a3998loq7v9e01
batch_06gcclodi3fqtidru2fmj8bt01
529
2026-09-21T23:40:42Z (status: COMPLETED, 529 run ids listed)
No — frozen until cancelled 2026-09-22T01:14:01Z
All three parents were created at 2026-09-21T23:28:33Z from one batchTriggerAndWait of 20 items (batch_06gccj5sinpguml4d39lrfi101) issued by the cron task social-metrics-collection run run_06gccic2vbemcv2j2okvr9t801. Cancelling the three frozen parents (Canceled by user, 01:14–01:19Z) is what finally let the grandparent's batch wait resolve.
Reproduction
We cannot reproduce on demand. It happened to 3 of 20 concurrent parents doing the same thing: await childTask.batchTriggerAndWait(items) with 470–580 items per batch, each item carrying an idempotency key and a small JSON payload. Parent task has maxDuration: 300, child maxDuration: 600, each on its own queue with a concurrency limit.
Expected behavior
Once every run in the batch is in a final state the batch should be marked complete and the waiting parent should be resumed (or, if the batch record cannot be sealed, the parent should fail with an error rather than wait forever).
Additional information
Happy to share more run ids or grant dashboard access to the project if that helps.
Activity