Skip to content

gates: check-issue-citations retries a transient 5xx or connection reset on a board read before refusing - #23438

Merged
objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23361-citations-5xx-retry
Oct 14, 2026
Merged

objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23361-citations-5xx-retry

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #23361
Clause-②: no

What this changes

scripts/check-issue-citations.mjs read ONE HTTP 500 from the board as PREREQUISITE NOT MET (exit 3) with no retry, so a GitHub blip turned the lint lane red and ejected a PR from the merge queue (run 38496246256). Every board read now goes through one helper, readBoard, shared by both strategies (probeBoard's head probe and per-number probes, enumerateBoard's pages), with ONE retry budget per run created in buildBoard.

  • Retried (closed set): an HTTP 5xx; a thrown connection failure whose cause chain carries ECONNRESET or UND_ERR_SOCKET (RETRYABLE_NETWORK_CODES).
  • Not retried: every 4xx (a 404 keeps its meaning, "absent"; a 401/403/429 is an answer about the credential or the rate limit), and every other thrown error (ECONNREFUSED, ENOTFOUND, a proxy refusing the tunnel, a body that is not JSON). Those are rethrown unchanged, so they keep the exit path they had.
  • Never a weaker gate: a read still failing after its ladder is handed back as a failure; run()'s existing catch turns it into prerequisiteRefusal, exit 3, never 0. The refusal now names the read, the answer and the attempt count, for example GET issues?per_page=1 -> HTTP 500 after 3 attempts (every attempt one read is allowed, so not one blip).
  • The retried unit is the request AND its JSON body. A reset can land while the body streams; measured, res.json() then throws TypeError: terminated with the same cause codes.
  • run() and buildBoard take optional fetchImpl / sleep (defaults: the global fetch, a real setTimeout sleep). Only the self-test passes them.
  • makeTransferProbe deliberately does not share the retry. Its 3xx is a verdict and stays one. A failed read there never exits 3: it degrades to a null verdict, which classifies as allocated-but-absent, still a finding. It is only reached under --probe-cause, which no workflow runs.
  • The header docblock and line 204 (dispatch-gates: local-env, pinned by line number in dispatch-gates.self-test.mjs) are untouched. All edits sit below it.

Retry policy and worst-case added wall time

BOARD_READ_RETRY: 3 attempts per read (one read plus two retries), backoff 500 ms then 1000 ms, and 4 retries per run shared by every read the run makes.

board added sleep extra requests then
answers normally 0 0 unchanged
stays down (every read 5xx or reset) 1500 ms 2 exit 3
flaps (every read fails twice, then answers): the worst case at most 3000 ms at most 4 the next 5xx is final, then exit 3
  • Board that stays down: the first read (the head probe, or page 1 under --enumerate) spends its ladder and the board build ends, so the cost does not depend on how many numbers the change cites. The self-test pins it with 400 cited numbers (the most probeBoard takes before switching to enumeration) and with the enumerating strategy: 3 reads and 1500 ms in both.
  • Board that flaps: the per-run budget is what bounds a 400-number probe or a 159-page enumeration. Pinned: sleeps 500, 1000, 500, 1000 (3000 ms total), then ... after 1 attempt (this run's budget of 4 retries is spent).
  • Before this change a board that stays down cost 1 request and 0 ms; now it costs 3 requests and 1.5 s. Request latency itself is not bounded, before or after: no read in this file sets a timeout.

Which connection-failure shapes are retried, as measured

Measured on node v22.22.0 (CI runs node 22) against a local socket server, both directly and through a local CONNECT proxy under --use-env-proxy (the gate's own re-exec route):

what the peer does what fetch throws retried
reset (RST) after the request bytes, or on accept TypeError: fetch failed, cause code ECONNRESET yes
close (FIN) with no response fetch failed, cause code UND_ERR_SOCKET ("other side closed") yes
reset mid-body TypeError: terminated from the body read, cause ECONNRESET yes
close mid-body terminated, cause UND_ERR_SOCKET yes
tunnel reset or closed by the proxy fetch failed, cause ECONNRESET / UND_ERR_SOCKET yes
proxy answers CONNECT with 502/503 fetch failed, cause chain ending in UND_ERR_ABORTED no
connection refused (control) cause ECONNREFUSED no
DNS failure (control) cause ENOTFOUND no

Verification (head 3becc14a)

  • Self-test: node scripts/check-issue-citations.mjs --self-test, exit 0, (215 cases, 10 batteries). It was 173 cases and 9 batteries. The new retry battery has 42 cases (floor 42), and SELF_TEST_BATTERY_FLOOR goes from 9 to 10.
    • The card's cases: one transient 500 then a 200 resolves, on the head probe and on a per-number probe. Persistent 500s are refused after exactly 3 attempts, and exit 3 end to end, read off a real child process. That child is offline: its fetchImpl is a stub, and it judges a real temp-repo diff. A 404 is asked once, never retried, and is still a finding.
    • Also: reset then 200, and body-stream close then 200, resolve. A transient 502 on enumerate page 2 still yields the whole board. 401/403/429 are refused on the first attempt. Unnamed errors are rethrown as the identical error object. The worst-case wall readings are pinned as above. A table of 11 persistent failures (500, 502, 503, 504, ECONNRESET, UND_ERR_SOCKET, UND_ERR_SOCKET mid-body, 401, 403, 429, ECONNREFUSED) is applied at the head probe, at a per-number probe and at an enumerated page: none yields a board.
  • Battery wall: 5 interleaved runs each on this shared box. Before (the base file, blob c732ce2e, read from a checkout at 4f81f315): median 5418 ms. After: median 5722 ms. The +0.3 s is the offline child process and its temp repo. Backoff sleeps are injected, so the new cases add no sleep.
  • Reverse verification (one-time, no permanent file): each mutation went through scripts/ablation-replace.mjs in WRAP mode, at HEAD 3becc14a. Each restore was proven by blob == HEAD (d2a21576) and an empty git diff HEAD. Every leg went red by name:
    • A1 attemptsPerRead: 1 (no retry): red at "one transient 500 on the HEAD probe, then a 200, RESOLVES ... it refused instead: GET issues?per_page=1 -> HTTP 500 after 1 attempt".
    • A2 run()'s board-read catch returns 0 instead of refusing: red at "persistent 500s EXIT 3 end to end, never 0 — got exit 0".
    • A3 4xx made retryable: red at "a 404 keeps its meaning: asked ONCE, never retried".
    • A4 run budget deleted: red at "a board that FLAPS spends at most the run's 4 retries (3000 ms)", which slept 500, 1000 six times.
    • A5 an exhausted read reported as a board: red at "persistent 500s are NOT a board".
  • Real-process readings (not kept as tests):
    • Live board, --base 4f81f315~15: 60 citations across 33 files, probed (16 citations), frontier #23434, exit 0. This is readBoard over real responses.
    • Through a local proxy that resets every tunnel: exit 3 after 3 CONNECTs and 2.7 s of wall, refusal GET issues?per_page=1 -> ECONNRESET after 3 attempts (...).
    • Through a proxy that answers CONNECT with 503: exit 3 after 1 CONNECT, not retried, refusal fetch failed as before.
  • This gate on this diff: node scripts/check-issue-citations.mjs, exit 0: no issue citations added against 4f81f3152 (0 file(s) read). The diff is all under scripts/**, a deferred surface.
  • Derived gate union at 3becc14a: dispatch-gates --commands --repo objectstack-ai/objectstack derived 32 commands. All 32 ran and all exited 0. --ran with exit codes reconciles: "32 derived famil(ies) accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero)".
    • That includes pnpm check:pm-dispatch-gates (fast tier, 1841 cases), whose census pin scripts/check-issue-citations.mjs:204 local-env and live local-env rows for this script both pass.
    • The battery's slow tier (6 sections) is NOT MEASURED locally. Its home is push and schedule runs.
  • Lint, narrowed: eslint --no-inline-config --format json scripts/check-issue-citations.mjs gave 1 file, 0 errors, 0 warnings.
    • Population: eslint.config.mjs's **/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs} object covers the file.
    • Invariance: the resolved config's parserOptions are ecmaVersion and sourceType only, with no project and no projectService. Type-aware linting is off, so this diff cannot move another file's verdict.
    • Full pnpm lint is CI's run.
  • Declared to CI, not run here: 4 workflow-valued families (--census among them), 11 wide-population families, 55 artifact-roster families, the path-scheduled Test Core job and the type-check lanes. No package is touched.
    • The 15 pending-changeset families do not apply: this PR carries skip-changeset, because root scripts/ publishes nothing.

Acceptance notes

  • Under --probe-cause, a transfer probe that reads a 5xx yields a null verdict, whose detail text says "re-run with --probe-cause" even though the run already passed it. That is report wording, in a mode no workflow runs; it never passes a board. Noted, not filed. Carrier: none.
  • No board read sets a request timeout, so a hung read waits out the job timeout. This predates the change and is unchanged. It was not observed in the incident. Noted, not filed.
  • A thrown error outside the retried set still surfaces as a bare fetch failed, which does not name the read. It is kept byte-identical so those exit paths are unchanged.
  • scripts/check-shard-attestation.mjs carries its own githubFetchJson (one retry on 5xx, 429 or any network error). It was not reused: this ruling excludes 429 and unnamed network errors.

Generated by Claude Code

…re refusing

A 5xx, an ECONNRESET or an UND_ERR_SOCKET from a board read now gets a
bounded ladder (3 attempts per read, 500/1000 ms backoff, 4 retries per
run) before the gate concludes PREREQUISITE NOT MET. A 4xx and every other
thrown error are answers and are not retried; a read still failing after
its ladder keeps its exit-3 path, and the refusal names the attempt count.

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
A new `retry` battery (42 cases, floor raised to 10 batteries): one
transient 500 then a 200 resolves on the head probe and on a per-number
probe; persistent 500s are refused after exactly the ladder and exit 3 end
to end; a 404 is asked once and stays a finding; every other 4xx and every
unnamed thrown error is not retried; the per-run budget caps a flapping
board at 3000 ms of backoff; no scripted failure at any read yields a board.

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
…en it refuses

Every resolve case settles its promise, so a regression reds that case by
name rather than escaping as an uncaught throw; the per-read refusal reads
"every attempt one read is allowed, so not one blip".

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 38538817058 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

分类:failure —— 按下面的日志分诊。

失败的 job(日志抽取,best effort):

  • Build Core — 失败步骤: Build packages (excluding docs)(日志不可读,点进 job 看)
  • Test Core (2/6) — 失败步骤: Build this shard's dependency closure(日志不可读,点进 job 看)

↳ 失败原因 是判读的关键:超时(Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言(AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

⚠️ 断言这一侧有一类例外,判据是断言在测什么,不是它是不是 AssertionError。 断言的对象是产品行为(一个值、一个形状、一次拒收)⇒ 照上面读:真实的行为改变,去查,⛔ 不要重排掉;
断言的对象是这次实验自身的有效性前提(跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件)⇒ 它跟超时是同一类,同样对负载敏感,重排一次是合法的判别手段。
识别是机械的:断言的消息或它比较的值本身点名了一段时长、一个时间戳、一个耗时计数。实测过的一对 —— AssertionError: SecurityPlugin.init() ran: expected false to be true 测的是产品行为(真回归);
AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000 测的是实验前提:它守护的那条不变式当时是绿的,同一个 head 原样重排一次即成功。
穿着 AssertionError 外衣的时间测量,仍然是时间测量。(⛔ 这只改「怎么读一次红」,不改「哪些测试可以重排」——后者由别处管。)

跨 PR 相同签名(24h,按失败测试文件聚合):

  • ⚠️ 本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)—— 这不是「没有同签名的其他 PR」,是这一轮没测到。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 5 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Queue red, disposition · domain:devx#2 · session_01B7mrh59UAGffd7s9asUfoY · 2026-10-14T09:13Z. Answers the merge-queue triage comment 6164964851.

Not this PR's. No re-queue. This PR changes only scripts/check-issue-citations.mjs, which is part of no package build.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants