Skip to content

ci: prune each Test Core Turbo seed to its own run's tasks before saving it - #25523

Merged
os-zhuang merged 1 commit into
mainfrom
claude/project-thread-53z7nt
Oct 24, 2026
Merged

os-zhuang merged 1 commit into
mainfrom
claude/project-thread-53z7nt

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Elon · project thread

Fixes #25452

Clause-②: no
Metadata-protocol: none

Before each Test Core shard saves its Turbo seed on a push to main, the seed is now pruned to the cache entries its own run used. The Test Core seeds were not 2.4 GB of useful cache. They were one seed lineage that grew by one run per merge and was never trimmed. Pruning fixes this at the source, so the card's change (b) is the only change here. Change (a), dropping the Build Core / Type Check seeds, and change (c), the Next.js cache, are left alone: the measurement below shows those seeds are 0.05 GB each, not where the pool goes.

Step 1: the measurement (save-step and restore-step logs; the pool API is refused from the cloud container)

Bytes saved per merge (push run 39982654692, main at 05cc3ee953, a completed run; read from each save step's Sent N of N (100.0%) line):

seed key prefix per seed seeds per merge per merge
Linux-turbo-test-<1..6>-main- 2.410 – 2.441 GB 6 14.60 GB
Linux-turbo-build-core-main- 0.048 GB 1 0.048 GB
Linux-turbo-dogfood-<1..3>-main- 0.048 GB 3 0.145 GB
Linux-turbo-typecheck-main- (lint.yml run 39982654477) 0.054 GB 1 0.054 GB
total ≈ 14.85 GB (Test Core = 98 %)

Each merge writes about 1.5× the repository's 10 GB default pool. The jobs API shows every one of the six Test Core save steps succeeding on five more push runs from 16:23Z to 17:56Z (39986349408, 39989699728, 39994571067, 39996442855, 39999263178), including the cancelled runs.

Eviction, seen from a PR restore. On PR run 39999049930 (17:55Z), Test Core (1/6) (job 120570291429) restored Linux-turbo-test-3-main-fa06169c… (2.50 GB). That is shard 3's seed, reached through the bare Linux-turbo-test- restore key. The push at 17:36Z (run 39996442855) had saved a shard-1 seed, but no Linux-turbo-test-1-* entry was left 19 minutes later.

Why 2.4 GB: the seed snowballs. A full 76-task workspace build is 0.048 GB in Build Core's seed, and a shard's closure is a subset of that build. The restore step takes the newest seed by prefix, turbo never evicts its local cache, and the save step stored the whole directory. So every seed was the previous seed plus one run, and the bare-prefix fallback carried the lineage from shard to shard. That is why all six shards are the same size. The sizes keep growing: shard 3 was 2.441 GB at 05cc3ee9 (16:20Z) and 2.50 GB at fa06169c (17:36Z). The same growth reproduces on a two-package turbo 2.11.5 workspace: one input change took .turbo/cache from 9 files to 18, and pruning to the run summary took it back to 9 with a full 3 cached, 3 total replay.

Pool total and largest entries: pending. GET /actions/cache/usage and /actions/caches answer 403 from a cloud container ("Access to this GitHub Actions path is not permitted through this proxy"). The devx seat has asked the maintainer for the two reads, and they will be added here when they arrive.

What changed

  • scripts/ci/prune-turbo-cache.mjs (new). It reads every run summary (--summarize) in --run-summaries, plus any --extra-summaries, and collects each task's hash. It then deletes every <hash>.tar.zst / <hash>-meta.json / <hash>-manifest.json in .turbo/cache whose hash no summary names, and appends save=true|false to GITHUB_OUTPUT.
    • No readable run summary means save=false. That covers a cancelled test step or an empty shard. An extra summary alone (the closure build's) does not count, because saving builds without test results would make the newest seed one that replays no tests.
    • A file in any other shape is kept and reported, never deleted, so a turbo layout change shows up as a count and does not empty the cache.
    • The script exits 0 on every run-time path, so it cannot fail the shard or withhold its attestation. Malformed arguments exit 3.
    • --self-test covers 25 cases across 5 batteries with a pinned battery floor and the verdict handshake.
  • ci.yml, Test Core: a new step, Prune the Turbo cache to this run's tasks (main only) (if: always() && github.event_name == 'push'), runs before Save Turbo cache (main only). It reads .turbo/runs (the test step's summary) plus $RUNNER_TEMP/turbo-remote-cache (the closure build's summary, moved there by the remote-cache assertion). The save step's if: gains && steps.prune-turbo-cache.outputs.save == 'true'.
  • ci.yml, Build Core: a new prune-turbo-cache self-test step next to assert-turbo-remote-cache self-test, and the script is added to the core filter's guard rows (check-ci-filter-parity and check-self-test-wired both require it).

Kept on purpose: the bare Linux-turbo-test- restore key. Once seeds are pruned, a cross-shard restore brings at most another shard's pruned seed, about one closure's worth. The shared build entries still replay, and the prune removes the rest before the next save, so the fallback can no longer carry a lineage forward. Removing it would also take away the warm start its comment describes for a new shard number.

Expected after

Each Test Core seed should hold its shard's closure builds plus its test results, which is well under Build Core's 0.048 GB full build plus test logs. A merge would then write about 0.3–0.6 GB in total instead of about 14.85 GB. A side effect: PR shards stop downloading about 2.4 GB in Restore Turbo cache, which measured 15–46 s per shard on the PR runs above. The first push after landing still restores the old snowball once, prunes it, and saves the small seed. The old 2.4 GB entries then age out of the pool.

Acceptance (from the card)

  • Before: the per-merge table above. The pool total and its four shares will be added when the maintainer's gh api reads arrive.
  • After, owed after ≥ 5 merges to main post-landing: the Test Core save-step sizes, the pool total (target ≤ 8 GB at steady state), and two readings an hour apart. Over the next 10 PR runs whose closure matches main: the median of the shard "Build this shard's dependency closure" step ≤ 1.7 min and Build Core's build step ≤ 2.5 min. The assert-turbo-remote-cache hit counts will be recorded before and after.
  • ⛔ The pool limit is not raised here. If the pool is still over budget after this lands, the billing option goes back to the maintainer with the numbers.

Risks and rollback

  • Test replay on main depends on this seed. test / test:repo are off the remote cache (Build Core's PINS), so a pushed shard replays unchanged packages' tests only from the Actions seed. The prune keeps every hash the test step's summary names, which covers every test task and every build it depended on, so the replay set is unchanged. A test task that failed has no entry either way.
  • A shard whose test step left no summary saves nothing. The previous seed stays the newest, and the next completed push writes a fresh one.
  • Rollback: revert this PR. It touches no package and changes no restore key.

Local verification (head fcae6e3)

  • node scripts/ci/prune-turbo-cache.mjs --self-test: OK, 25 cases across 5 batteries. Firing proof: with the keep rule ablated to keep everything (a temporary copy of the script), the self-test exits 1 with 5 failures (every file of an unnamed hash is dropped, an inherited entry no summary names is deleted, …). The copy was deleted afterwards.
  • A real turbo 2.11.5 workspace: prune to the run summary, delete dist/ and the summaries, re-run, and the result is Cached: 3 cached, 3 total with outputs restored.
  • All exit 0: check-self-test-wired (+ --self-test), check-ci-filter-parity (+ --self-test), check-step-collectors (+ --self-test), check-self-test-workflow-commands, check-published-list-mirrors, check-declaration-mirrors, check-platform-checklist-watchdog, check-empty-changeset --base origin/main, check-adr-0087-registration --base origin/main, check-aggregator-roster, check-closing-keyword-parity, check-comment-mask-corpus, check-shard-attestation --self-test, and pnpm check: workflow-status-functions, workflow-step-name-quoting, nul-bytes, merged-result, pm-governed-merges, pm-expected-skips, single-claim-paths, pnpm-filter-targets, pnpm-flag-spellings, scripts-symbol-anchors, runner-env-posture, node-version, pnpm-acquisition, select-shard-packages, shard-attestation, required-contexts, stall-guard-budget, stall-guard-headroom, issue-citations, optional-error-sink, doc-authoring. The rest of the farm is left to CI.
  • ci.yml parses (yaml); eslint is clean on the new script.
  • check-governed-merges.mjs --test .github/workflows/ci.yml scripts/ci/prune-turbo-cache.mjs: GOVERNED, landing tier H (.github/workflows/** ×1), 331 changed lines.
  • skip-changeset: no published package is touched (.github/workflows/**, scripts/ci/**).

Acceptance notes

  • Branch name. The branch is claude/project-thread-53z7nt, the name the session was given. The claim (comment 6371859957) names it.
  • Card changes (a) and (c) not taken. Measured at 0.05 GB per seed, they are not where the pool goes. Revisit them only if the after-reading still exceeds 8 GB.

维护者速读(草稿)

改了什么:main 每次合并后,Test Core 的 6 个分片会各存一份 Turbo 缓存种子。现在存之前先把种子裁剪到「这次运行真正用到的任务」,其余旧条目删掉。如果测试步骤没有跑完(被取消或空分片),这次就不存。

为什么改:实测每份种子 2.4 GB,每次合并共写约 14.6 GB,而仓库的 Actions 缓存池只有 10 GB,结果是每合并一次,上一次的种子就被挤掉。原因是种子在滚雪球:每次都在恢复的旧种子上再加一轮,从不清理。完整构建一次的产物其实只有 0.05 GB。裁剪后预计每次合并只写 0.3–0.6 GB,PR 运行也不用再下载 2.4 GB 的种子。

风险与代价(含回滚):main 上测试结果的复用只靠这份种子,所以本次运行用到的条目(含全部测试结果)一律保留,复用范围不变。没跑完的运行不存种子,旧种子继续可用。回滚只需 revert,不动任何包和恢复键。缓存池总量的读数还在等,拿到后补进正文。

席位意见:同意落地(devx 席位已在 head fcae6e3 复核,ACCEPT 见 #25452)。日志已证实种子在滚雪球:每次合并约 14.6 GB,远超 10 GB 的缓存池。修剪只保留本次运行用过的任务,失败模式偏保守:认不出的文件保留,没有运行摘要就不存。改动前的缓存池总量等你本机读出后补进正文,不影响审批。

你要做的:本 PR 改了 .github/workflows/**,属于 Tier H,需要 GOVERNED_APPROVERS 中的一位批准后,席位才会翻 ready 并落地。


Generated by Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01SawdJgiuoz8deqVdM51jAE

…ing it

The Test Core restore takes the newest seed by key prefix and turbo never
evicts its local cache, so every saved seed was the previous seed plus one
more run, and the bare `<os>-turbo-test-` fallback carried that lineage from
shard to shard. On push run 39982654692 each of the six shard seeds was
2.41-2.44 GB (Build Core's full 76-task build: 0.048 GB), about 14.6 GB per
merge into the repository's 10 GB Actions cache pool.

scripts/ci/prune-turbo-cache.mjs keeps only the cache entries named by the
test step's run summary and the closure build's summary, drops the rest,
and reports save=false when the test step left no summary (a cancelled run
or an empty shard) so such a run no longer saves a seed. The save step
reads that output. The self-test runs once in Build Core.

Claude-Session: https://claude.ai/code/session_01SawdJgiuoz8deqVdM51jAE
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Turbo seeds vs the 10 GB Actions cache pool — measure the pool, then shrink or retire the per-sha seeds

3 participants