Skip to content

feat(fetch): retry uncharged busy answers so a brief capacity wait does not fail the command - #64

Open
gonzalo-zenrows wants to merge 6 commits into
mainfrom
feat/fetch-retry-jitter
Open

gonzalo-zenrows wants to merge 6 commits into
mainfrom
feat/fetch-retry-jitter

Conversation

@gonzalo-zenrows

@gonzalo-zenrows gonzalo-zenrows commented Oct 16, 2026 •

Copy link
Copy Markdown

Why

zenrows fetch and zenrows extract sent each request once. A brief busy answer (429, 502, 503 or 504) failed the command, although Zenrows does not charge these answers. A 503 RESP009 told agents to rerun with --manual --js-render --premium-proxy. That is the most expensive configuration, and it does not help.

This PR applies the retry policy that every Zenrows Fetch and Extract client shares.

What changes

  1. Retried statuses. The CLI retries 429, 502, 503 and 504 up to --retries <n> times. The flag is new on fetch and extract. The default is 2, and 0 sends the request once.
  2. Retried network errors. The CLI retries a network error only when the request never reached the API. That means a refused connection, a name that did not resolve, a connect timeout or an unreachable network.
  3. Connect timeouts. A host with several addresses gets one connect per address family. The CLI retries only when every connect failed, so the request was never sent. A connect timeout (ETIMEDOUT on connect) counts as such a failure, but a read timeout never does.
  4. Not retried. The CLI never retries its own timeout, or a connection that closed after the request was sent. In both cases the API may still finish the request and charge it.
  5. Target statuses. Under --original-status, or when allowed_status_codes lists the status, the 429 or 5xx may be the target site's page. The CLI sends it once.
  6. Delay. Retry-After counts when it is whole digits. The CLI waits that many seconds, at least 0.25 s and at most 30 s. Otherwise it waits uniform(0.25 s, min(30 s, 1 s × 2^attempt)), where attempt counts retries from 0.
  7. Budget. No retry starts after 90 s, counted from the command's first request. The budget does not stop a running request, so a command can take longer than 90 s.
  8. Timeout. The default --timeout rises from 120 s to 200 s. The API's own budget is 180 s, and the public docs tell clients to wait at least that long. A test again guards that the default stays above the budget.
  9. SERVICE_BUSY. A 503 RESP009 now returns this new error. It says the request was not charged. It quotes the API's own title, which tells the two Extract meanings apart. The next action says to wait and run the same command again.
  10. The same command. SERVICE_BUSY and the final error after a retried 429, 502, 503 or 504 suggest the command that ran. The suggestion keeps the extract method and every flag, quoted for a shell. These errors no longer advise --js-render or --premium-proxy.
  11. Attempts. With --json, the result has a new attempts field. It counts the HTTP requests the command sent, so 1 means no retry.
  12. extract prepare is not retried. Its endpoints answer 429 when the monthly allowance is used up and 503 when preparation is unavailable. Those are answers, not busy signals. The command keeps its own client and sends each request once, and a test guards that.
  13. Shared helper. src/core/backoff.ts holds the delay schedules. Crawl keeps its own schedule and its own reading of Retry-After, so its behaviour does not change.
  14. Docs. The fetch and extract help, the README Fetch section, and the protected-fetch, trace-debug and extract skills describe all of the above.

Behaviour change

Agents that branched on FETCH_FAILED for a 503 RESP009 now see SERVICE_BUSY. A command that hits a busy API now takes longer before it fails, but no retry starts after 90 s. A slow request now waits up to 200 s before the CLI gives up, instead of 120 s. A connection that closes after the request was sent now fails at once instead of being retried. The --json result gains an attempts field, and no existing field changes. Exit codes stay the same. zenrows extract prepare behaves as it does on main.

Not covered

A 429 page that is not JSON and mentions "upgrade" or "credit" is now retried. After the retries, the existing quota check still reports it as POLICY_MAX_CREDITS_EXCEEDED. That check also serves zenrows usage, so changing it belongs in a separate PR. Under --original-status, a RESP009 from Zenrows itself is now sent once, because the CLI cannot always tell it from a target page. Error envelopes other than --json results do not report the attempt count. The CLI has no cancel option of its own; Ctrl-C ends the command, including a wait between retries. I did not run any of this against the live API.

Testing

  1. npm ci, npm run typecheck and npm test pass on the branch head: 588 tests, 0 failures, on Node 26 locally. That count includes every test from main, among them the extract prepare tests. CI runs Node 20 and 22.
  2. tests/fetch-retry.test.ts holds the shared conformance tables for Retry-After, jitter bounds and the retry decision. It also checks the 90 s budget on both paths and exact waits with a fixed random source.
  3. Three tests use real node:http sockets. A refused port gets three attempts, on 127.0.0.1 and on localhost. A server that closes the connection after reading the request gets one.
  4. tests/fetch-retry-command.test.ts runs the fetch and extract commands. It checks the suggested command for each extract method and a manual fetch. It also checks --retries 0, the attempts field, and that extract prepare sends a 429 or 503 once.
  5. I ran the built CLI against a local mock API. A 503 RESP009 on fetch and on extract --css ... --json made three requests each and suggested the exact command. extract prepare and extract prepare status made one request per endpoint on a 429 or 503.
  6. I reverted each change in turn on a copy of the tree. Every mutation made at least one test fail.

…es not fail the command

Fetch and Extract sent each request once. A 429, 502, 503 or 504 (none of
which the gateway charges) or a network error failed the command, and 503
RESP009 fell into the generic branch that advised the most expensive manual
configuration.

- Retry those answers up to --retries times (default 2; 0 sends once). Each
  wait honours Retry-After in whole seconds, capped at 30 s, or is full
  jitter: uniform(0.25 s, min(30 s, 1 s x 2^attempt)). No retry starts once
  the time spent plus the next wait would pass 90 s.
- Never retry the client timeout: the API keeps working and may still charge
  the request. Never retry other 4xx, or a 429 that says the account is out
  of credits. The Extract-not-enabled fallback to autoparse is unchanged.
- 503 RESP009 is now SERVICE_BUSY: not charged, retry shortly, no escalation.
- The gateway budget is 180 s for the whole request, not 90 s. The client
  timeout stays at 120 s; the timeout error now says the API may still
  finish and charge the request, and suggests a value above the budget.
- Retry-After parsing and the delay schedules move to src/core/backoff.ts;
  Crawl keeps its own schedule unchanged.
…back to the command they ran

The retry change resent requests the API may already be working on, retried
pages the target site sent, and told agents to run a different command than
the one that failed.

- Retry a network error only when the request never reached the API
  (refused connection, DNS failure, connect timeout, unreachable network).
  A connection that closes after the request was sent is not retried: the
  API may still finish and charge it, as with a timeout.
- Do not retry a 429, 502, 503 or 504 under original_status, or when
  allowed_status_codes covers it: that status may be the target site's page.
- Decide a 429 by its error code: AUTH004 and AUTH014 are not retried, any
  other 429 is, including a non-JSON page.
- Read Retry-After as whole digits clamped to 0.25-30 s, so 0 cannot drive a
  tight loop. Crawl keeps its own reading. --retries takes whole decimal
  digits only, so 1e1, 0x2 and padded values are usage errors.
- Extract's Autoparse fallback shares the first request's 90 s budget.
- Raise the default client timeout to 200 s, above the API's 180 s budget,
  as the public docs advise, and restore the test that guards it.
- SERVICE_BUSY and the final error after a retried 429, 502, 503 or 504
  suggest the command that ran, with its flags and extract method, and no
  longer advise the most expensive configuration. On Extract, SERVICE_BUSY
  quotes the API's own title, which says which of RESP009's two meanings
  applies.
- --json results report attempts, the number of requests sent.
@gonzalo-zenrows
gonzalo-zenrows force-pushed the feat/fetch-retry-jitter branch from 0df80be to 4104b09 Compare October 19, 2026 21:03
@gonzalo-zenrows
gonzalo-zenrows marked this pull request as ready for review October 20, 2026 08:11
@gonzalo-zenrows gonzalo-zenrows self-assigned this Oct 20, 2026

@jonathan-zenrows jonathan-zenrows left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I'd merge after the rebase (it conflicts with main). Checked: typecheck clean, 505/505 tests pass, and the retry notices go to stderr, so piped --json stays clean.

Two things go beyond what the other 4 PRs do. We should decide whether all of them get them or this one drops them:

  • a 429 with AUTH004/AUTH014 is not retried (src/core/http.ts:89, 219)
  • with --original-status or an allowed_status_codes that covers the status, nothing is retried (http.ts:227-234). That includes a real 503 RESP009 from ZenRows. If you keep it, I'd still retry when the body is a ZenRows error with a retryable code.

Small:

  • with happy-eyeballs, a connect timeout on one address family arrives as an AggregateError with code: ETIMEDOUT. It isn't in NEVER_SENT_CODES, so it isn't retried. It's a connect, so it's safe to retry: check syscall === 'connect'.
  • clearTimeout runs before res.arrayBuffer() (http.ts:309-314), so reading the body has no timeout, and a failure mid-body throws a raw TypeError instead of a ToolkitError. This was already in main, but the docblock at l.270 now says the opposite.
  • the comment in src/adapters/protected-fetch.ts:48-51 should give the numbers (200 s vs 180 s) like http.ts does.
  • README §8 Extract doesn't mention --retries.

… as a 402

The retry decision skipped a 429 whose code was AUTH004 or AUTH014. The
API sends both codes as a 402, never as a 429. A 402 is not a retried
status, so the rule never ran and only made the policy harder to read.

A 429 is now retried like any other 429. A status the target site sent,
under original_status or allowed_status_codes, is still sent once. The
out-of-credits error that a 402 produces is unchanged.

The tests now prove that a 429 with one of these codes is retried, and
that a 402 AUTH004 is sent once. The README and the protected-fetch
skill no longer describe the rule.
When a host has several addresses, Node tries one connect per address
family. If they all fail, fetch rejects with an AggregateError. A connect
timeout on one family carries the code ETIMEDOUT with the syscall connect.
That code was not in the never-sent list, so the CLI sent such a request
once, although a failed connect never wrote the request.

Node also copies the first inner error's code onto the AggregateError.
The check read that copied code first, so a refused connect followed by a
reset still counted as never sent. The inner errors now decide: every one
of them must be a connect failure, and an empty list does not count.

An ETIMEDOUT now counts only when its syscall is connect. A read timeout
is still sent once, because the API may still finish and charge it.
The comment on one Fetch attempt said its timeout also covers the body
read. It does not: the timer is cleared before the body is read, so a
slow body has no timeout and a failure there is not a normalized error.
The comment now says that. The behaviour is unchanged.

The timeoutMs comment in protected-fetch now gives the numbers that
http.ts uses: 200 s by default, per HTTP attempt, above the API's own
180 s budget. It names both constants, so a reader can check them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants