Repository navigation
feat(fetch): retry uncharged busy answers so a brief capacity wait does not fail the command - #64
Open
gonzalo-zenrows wants to merge 6 commits into
Open
gonzalo-zenrows wants to merge 6 commits into
gonzalo-zenrows wants to merge 6 commits into
Conversation
This was referenced Oct 16, 2026
Open
Author
|
Same retry policy across all clients (429/502/503/504 and pre-response connection errors; 2 retries by default; full jitter; Retry-After capped at 30 s; 90 s total budget; timeouts never retried):
|
…es not fail the command Fetch and Extract sent each request once. A 429, 502, 503 or 504 (none of which the gateway charges) or a network error failed the command, and 503 RESP009 fell into the generic branch that advised the most expensive manual configuration. - Retry those answers up to --retries times (default 2; 0 sends once). Each wait honours Retry-After in whole seconds, capped at 30 s, or is full jitter: uniform(0.25 s, min(30 s, 1 s x 2^attempt)). No retry starts once the time spent plus the next wait would pass 90 s. - Never retry the client timeout: the API keeps working and may still charge the request. Never retry other 4xx, or a 429 that says the account is out of credits. The Extract-not-enabled fallback to autoparse is unchanged. - 503 RESP009 is now SERVICE_BUSY: not charged, retry shortly, no escalation. - The gateway budget is 180 s for the whole request, not 90 s. The client timeout stays at 120 s; the timeout error now says the API may still finish and charge the request, and suggests a value above the budget. - Retry-After parsing and the delay schedules move to src/core/backoff.ts; Crawl keeps its own schedule unchanged.
…back to the command they ran The retry change resent requests the API may already be working on, retried pages the target site sent, and told agents to run a different command than the one that failed. - Retry a network error only when the request never reached the API (refused connection, DNS failure, connect timeout, unreachable network). A connection that closes after the request was sent is not retried: the API may still finish and charge it, as with a timeout. - Do not retry a 429, 502, 503 or 504 under original_status, or when allowed_status_codes covers it: that status may be the target site's page. - Decide a 429 by its error code: AUTH004 and AUTH014 are not retried, any other 429 is, including a non-JSON page. - Read Retry-After as whole digits clamped to 0.25-30 s, so 0 cannot drive a tight loop. Crawl keeps its own reading. --retries takes whole decimal digits only, so 1e1, 0x2 and padded values are usage errors. - Extract's Autoparse fallback shares the first request's 90 s budget. - Raise the default client timeout to 200 s, above the API's 180 s budget, as the public docs advise, and restore the test that guards it. - SERVICE_BUSY and the final error after a retried 429, 502, 503 or 504 suggest the command that ran, with its flags and extract method, and no longer advise the most expensive configuration. On Extract, SERVICE_BUSY quotes the API's own title, which says which of RESP009's two meanings applies. - --json results report attempts, the number of requests sent.
gonzalo-zenrows
force-pushed
the
feat/fetch-retry-jitter
branch
from
October 19, 2026 21:03
0df80be to
4104b09
Compare
gonzalo-zenrows
marked this pull request as ready for review
October 20, 2026 08:11
jonathan-zenrows
left a comment
Contributor
There was a problem hiding this comment.
Looks good, I'd merge after the rebase (it conflicts with main). Checked: typecheck clean, 505/505 tests pass, and the retry notices go to stderr, so piped --json stays clean.
Two things go beyond what the other 4 PRs do. We should decide whether all of them get them or this one drops them:
- a 429 with
AUTH004/AUTH014is not retried (src/core/http.ts:89, 219) - with
--original-statusor anallowed_status_codesthat covers the status, nothing is retried (http.ts:227-234). That includes a real503 RESP009from ZenRows. If you keep it, I'd still retry when the body is a ZenRows error with a retryablecode.
Small:
- with happy-eyeballs, a connect timeout on one address family arrives as an
AggregateErrorwithcode: ETIMEDOUT. It isn't inNEVER_SENT_CODES, so it isn't retried. It's a connect, so it's safe to retry: checksyscall === 'connect'. clearTimeoutruns beforeres.arrayBuffer()(http.ts:309-314), so reading the body has no timeout, and a failure mid-body throws a rawTypeErrorinstead of aToolkitError. This was already in main, but the docblock at l.270 now says the opposite.- the comment in
src/adapters/protected-fetch.ts:48-51should give the numbers (200 s vs 180 s) likehttp.tsdoes. - README §8 Extract doesn't mention
--retries.
… as a 402 The retry decision skipped a 429 whose code was AUTH004 or AUTH014. The API sends both codes as a 402, never as a 429. A 402 is not a retried status, so the rule never ran and only made the policy harder to read. A 429 is now retried like any other 429. A status the target site sent, under original_status or allowed_status_codes, is still sent once. The out-of-credits error that a 402 produces is unchanged. The tests now prove that a 429 with one of these codes is retried, and that a 402 AUTH004 is sent once. The README and the protected-fetch skill no longer describe the rule.
When a host has several addresses, Node tries one connect per address family. If they all fail, fetch rejects with an AggregateError. A connect timeout on one family carries the code ETIMEDOUT with the syscall connect. That code was not in the never-sent list, so the CLI sent such a request once, although a failed connect never wrote the request. Node also copies the first inner error's code onto the AggregateError. The check read that copied code first, so a refused connect followed by a reset still counted as never sent. The inner errors now decide: every one of them must be a connect failure, and an empty list does not count. An ETIMEDOUT now counts only when its syscall is connect. A read timeout is still sent once, because the API may still finish and charge it.
The comment on one Fetch attempt said its timeout also covers the body read. It does not: the timer is cleared before the body is read, so a slow body has no timeout and a failure there is not a normalized error. The comment now says that. The behaviour is unchanged. The timeoutMs comment in protected-fetch now gives the numbers that http.ts uses: 200 s by default, per HTTP attempt, above the API's own 180 s budget. It names both constants, so a reader can check them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
zenrows fetchandzenrows extractsent each request once. A brief busy answer (429, 502, 503 or 504) failed the command, although Zenrows does not charge these answers. A 503RESP009told agents to rerun with--manual --js-render --premium-proxy. That is the most expensive configuration, and it does not help.This PR applies the retry policy that every Zenrows Fetch and Extract client shares.
What changes
--retries <n>times. The flag is new onfetchandextract. The default is 2, and0sends the request once.ETIMEDOUTonconnect) counts as such a failure, but a read timeout never does.--original-status, or whenallowed_status_codeslists the status, the 429 or 5xx may be the target site's page. The CLI sends it once.Retry-Aftercounts when it is whole digits. The CLI waits that many seconds, at least 0.25 s and at most 30 s. Otherwise it waitsuniform(0.25 s, min(30 s, 1 s × 2^attempt)), whereattemptcounts retries from 0.--timeoutrises from 120 s to 200 s. The API's own budget is 180 s, and the public docs tell clients to wait at least that long. A test again guards that the default stays above the budget.SERVICE_BUSY. A 503RESP009now returns this new error. It says the request was not charged. It quotes the API's own title, which tells the two Extract meanings apart. The next action says to wait and run the same command again.SERVICE_BUSYand the final error after a retried 429, 502, 503 or 504 suggest the command that ran. The suggestion keeps the extract method and every flag, quoted for a shell. These errors no longer advise--js-renderor--premium-proxy.--json, the result has a newattemptsfield. It counts the HTTP requests the command sent, so 1 means no retry.extract prepareis not retried. Its endpoints answer 429 when the monthly allowance is used up and 503 when preparation is unavailable. Those are answers, not busy signals. The command keeps its own client and sends each request once, and a test guards that.src/core/backoff.tsholds the delay schedules. Crawl keeps its own schedule and its own reading ofRetry-After, so its behaviour does not change.fetchandextracthelp, the README Fetch section, and theprotected-fetch,trace-debugandextractskills describe all of the above.Behaviour change
Agents that branched on
FETCH_FAILEDfor a 503RESP009now seeSERVICE_BUSY. A command that hits a busy API now takes longer before it fails, but no retry starts after 90 s. A slow request now waits up to 200 s before the CLI gives up, instead of 120 s. A connection that closes after the request was sent now fails at once instead of being retried. The--jsonresult gains anattemptsfield, and no existing field changes. Exit codes stay the same.zenrows extract preparebehaves as it does onmain.Not covered
A 429 page that is not JSON and mentions "upgrade" or "credit" is now retried. After the retries, the existing quota check still reports it as
POLICY_MAX_CREDITS_EXCEEDED. That check also serveszenrows usage, so changing it belongs in a separate PR. Under--original-status, aRESP009from Zenrows itself is now sent once, because the CLI cannot always tell it from a target page. Error envelopes other than--jsonresults do not report the attempt count. The CLI has no cancel option of its own; Ctrl-C ends the command, including a wait between retries. I did not run any of this against the live API.Testing
npm ci,npm run typecheckandnpm testpass on the branch head: 588 tests, 0 failures, on Node 26 locally. That count includes every test frommain, among them theextract preparetests. CI runs Node 20 and 22.tests/fetch-retry.test.tsholds the shared conformance tables forRetry-After, jitter bounds and the retry decision. It also checks the 90 s budget on both paths and exact waits with a fixed random source.node:httpsockets. A refused port gets three attempts, on127.0.0.1and onlocalhost. A server that closes the connection after reading the request gets one.tests/fetch-retry-command.test.tsruns thefetchandextractcommands. It checks the suggested command for each extract method and a manual fetch. It also checks--retries 0, theattemptsfield, and thatextract preparesends a 429 or 503 once.RESP009onfetchand onextract --css ... --jsonmade three requests each and suggested the exact command.extract prepareandextract prepare statusmade one request per endpoint on a 429 or 503.