Make PrimeCaches the only Rust cache writer and remove sccache by ludfjig · Pull Request #1963 · hyperlight-dev/hyperlight

ludfjig · 2026-10-21T00:24:33Z

Makes PrimeCaches the only workflow that writes Rust caches, and removes sccache.

PR CI hit sccache rate limits (429/503), and caches on main kept growing until they filled the 10 GB repository quota. rust-cache keeps every dependency version it restores, so entries saved after builds on restored caches grow with each lockfile change.

Remove sccache. The jobs it served (fuzz, non-PIE guests, bench baseline) get rust-cache entries.
PrimeCaches looks up each key and builds from a cold target only on a miss. Every other workflow restores only.
Scheduled, Miri and Copilot jobs no longer write caches.
A final PrimeCaches job deletes superseded v1-rust-* and sysroot-* entries on main.
Bump rust-cache to v2.9.3 and start under the v1-rust prefix, so the first run replaces the grown v0-rust entries.

A cold PrimeCaches run on this branch saved about 8.6 GB in total. The largest entries are code-checks (1.4 GB Linux, 1.0 GB Windows) and the eight host entries (540–620 MB each).

Follow-ups: caches for Miri X64, Windows arm64, macOS and trace fuzz guests.

Easiest to review commit-by-commit

v2.9.3 fixes the cleanup that runs after a partial cache restore. It returned after the first directory entry, so artifacts older than a week were kept and saved again in the next cache entry. See Swatinem/rust-cache#375. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

sccache stores one GitHub Actions cache entry per compiled crate. Jobs hit the cache service rate limits (HTTP 429 and 503), which turned cache reads into slow misses. Its many small entries also share the 10 GB repository cache quota with the rust-cache entries and push them out. rust-cache restores whole target directories with one request per job, so it covers the same dependency builds. Later commits extend it to the jobs sccache served alone: fuzzing, the non-PIE guests and the benchmark baseline. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Copilot

🟡 Changes recommended

Cache cleanup and fuzz priming have unresolved issues that undermine cache freshness and quota control.

3 open issues

What changed in this PR

Centralizes Rust cache writes in PrimeCaches to limit cache growth and avoid sccache rate limits.

Changes:

Adds cold cache priming and superseded-entry cleanup.
Removes sccache and makes other cache consumers restore-only.
Adds benchmark baseline target reuse and fuzz build priming.

File	Description
src/hyperlight_ci/src/bench.rs	Supports a separate baseline target directory.
Justfile	Adds priming recipes and shares the guest sysroot.
.github/workflows/RustNightly.yml	Removes nightly caching and sccache.
.github/workflows/PrimeCaches.yml	Coordinates priming and cache cleanup.
.github/workflows/dep_run_examples.yml	Restores versioned caches without sccache.
.github/workflows/dep_miri.yml	Makes Miri caches restore-only.
.github/workflows/dep_fuzzing.yml	Adds fuzz cache priming.
.github/workflows/dep_code_checks.yml	Restricts cache writes to priming.
.github/workflows/dep_build_test.yml	Restores host caches without writing.
.github/workflows/dep_build_guests.yml	Adds guest priming and non-PIE caching.
.github/workflows/dep_benchmarks.yml	Seeds the baseline target from restored artifacts.
.github/workflows/Coverage.yml	Removes coverage caching and sccache.
.github/workflows/copilot-setup-steps.yml	Disables toolchain caching.
.github/actions/setup-sccache/action.yml	Deletes the sccache setup action.

_{Review effort: Balanced}

Coverage and RustNightly run on a schedule. RustNightly saved its Rust and sysroot caches on every run. Their entries serve only the next scheduled run, yet count against the 10 GB repository quota and evict the entries pull requests restore. Both jobs now build without a cache. CargoAudit saved a rust-cache entry through setup-rust-toolchain, whose cache input defaults to true. It needs no build cache, so the input is turned off. Miri saved its cache only from DailyArm64, under the arm64 key. Pull requests run Miri on X64, so they never restored that entry. Miri now only restores. Priming the X64 Miri cache is left to a follow-up. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

setup-rust-toolchain enables rust-cache by default. Copilot code review runs these steps on pull request branches and saves an entry of about 100 MB per branch, such as v0-rust-copilot-pull-request-reviewer-Linux-x64-aea9c2a7-2b9f5a70. Branch entries serve only that branch and evict the main entries every pull request restores. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

build-rust-guests-non-pie ran cargo hyperlight in target-non-pie to make a sysroot. That also built simpleguest a second time, about 12 s per job. The non-PIE build then compiled into target-non-pie/build, which rust-cache did not cover, about 11 s more. sccache served these builds. The non-PIE build now depends on build-rust-guests and uses its sysroot in target/sysroot. The toolchain and target are identical, so the sysroot is identical. rust-cache covers target-non-pie/build, and the sysroot cache holds only target/sysroot. The guests job builds nothing in the root workspace, so its target entry is dropped. Its lockfile no longer invalidates the guest cache either. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

bench-against builds the baseline commit in a worktree under target/baseline-worktree. Its target directory is empty in CI, so every pull request compiled the baseline from scratch. sccache served those builds. cargo ci bench takes --baseline-target-dir, passed to the baseline build and the ballast helper build as --target-dir. The benchmarks job copies the restored rust cache to a separate directory and passes it. A copy is needed because cargo fingerprints path packages by workspace path and mtime, so two checkouts cannot share one target directory. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

PrimeCaches built only the host library. Pull request jobs then compiled every test binary, dev-dependency and hyperlight-ci on top of the restored cache, and rust-cache never saved those from a pull request. just prime-caches builds what just test compiles and what cargo ci needs. It sets the cargo runner for the host target to a command that exits 0, so every test binary builds and none runs. Doctests are left out because rustdoc compiles them only when they run. Measured locally, a debug entry grows to 390 MB compressed and a release entry to 330 MB. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Every reusable workflow saved its rust-cache entry whenever it ran on main. Scheduled jobs such as DailyArm64, Fuzzing and DailyBenchmarks raced PrimeCaches for the same keys. Each writer built on a restored entry, and rust-cache keeps every dependency version it restores, so entries grew with each lockfile change. Code checks reached 1.7 GB against 676 MB for a clean build. PrimeCaches is the only writer. Its jobs look up their entry with lookup-only. On an exact hit they skip the build. Otherwise they build from a cold target and save, so an entry holds only what one lockfile needs. dep_build_guests and dep_code_checks take a prime_cache input for this. Priming skips C guests and uploads. cache-on-failure is dropped, so a failed build never saves a partial entry under the exact key. PrimeCaches never cancels a running run, so the newest commit always saves. Its concurrency group is per ref, so a manual run on a branch cannot replace one for main. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

PR fuzz jobs compiled every fuzz target with sccache. With sccache removed they need a Rust cache. PrimeCaches builds every fuzz target from a cold target and saves one entry under the key fuzz-X64. Trace targets build with the trace feature, matching `just fuzz`. PR and scheduled fuzz jobs only restore it. The fuzz workflow installs the pinned nightly before the cache step. rust-cache hashes installed toolchains, so a pin change changes the key and triggers a new prime. The new `just build-fuzz-targets` recipe builds each target alone because the trace targets do not compile without the trace feature. It fails if `cargo fuzz list` fails or lists nothing, so an empty build is never cached. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Rust cache keys end in a toolchain hash and a lockfile hash. Sysroot keys end in one hash. Every dependency or toolchain update leaves the old entry on main until GitHub evicts it after seven days unused. Those entries count toward the 10 GB repository quota and push out current ones. A final PrimeCaches job keeps the newest entry for each key without its hashes and deletes the rest. It runs even if a prime job fails, which keeps the old entry for that job's key. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

The v0-rust entries on main grew over many builds on restored caches. PrimeCaches skips the build on an exact key match, so it would keep those entries until the next lockfile change. Every rust-cache step uses the prefix v1-rust. The first PrimeCaches run builds every entry from a cold target. The old v0-rust entries expire after seven days unused. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

syntactically · 2026-10-21T18:54:35Z

I wonder why we added sccache to begin with. Was it doing anything for C guest builds or anything?

hyperlight-gh-bot · 2026-10-21T21:20:11Z

Benchmark Results

Measured commit: 3d16ce373ba6
Baseline commit: 5e2d3eb72e6b

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`743.20 ns` (➖ 1.02x faster)
`vec_bytes`	`579.89 ns` (➖ 1.02x faster)
		`372.41 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`141.95 ns` (➖ 1.00x slower)
`262144`	`522.11 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`82.09 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.73 ns` (➖ 1.00x slower)	`7.77 ns` (➖ 1.01x slower)	`7.90 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`93.37 µs` (➖ 1.00x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.05 µs` (➖ 1.06x slower)	`1.95 µs` (➖ 1.08x faster)
`262144`	`7.05 µs` (➖ 1.02x faster)	`7.00 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.23 µs` (➖ 1.03x faster)	`6.21 µs` (➖ 1.02x faster)
`8192`	`1.06 µs` (➖ 1.08x faster)	`1.06 µs` (➖ 1.01x faster)
`262144`		`26.92 µs` (➖ 1.02x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`839.73 ns` (➖ 1.02x faster)
`vec_bytes`	`847.58 ns` (➖ 1.35x slower)
		`745.24 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`163.44 ns` (➖ 1.01x faster)
`262144`	`620.93 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`88.38 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.40 ns` (➖ 1.00x faster)	`8.31 ns` (➖ 1.01x faster)	`8.41 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`53.36 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.52 µs` (➖ 1.01x faster)	`2.52 µs` (➖ 1.00x faster)
`262144`	`8.93 µs` (➖ 1.01x faster)	`8.89 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.39 µs` (➖ 1.06x faster)	`8.22 µs` (➖ 1.07x faster)
`8192`	`906.28 ns` (➖ 1.02x faster)	`871.30 ns` (➖ 1.05x faster)
`262144`		`34.42 µs` (➖ 1.03x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`933.65 ns` (➖ 1.01x slower)
`vec_bytes`	`715.82 ns` (➖ 1.01x slower)
		`319.26 µs` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`191.43 ns` (➖ 1.00x slower)
`262144`	`739.64 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`57.52 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.76 ns` (➖ 1.02x slower)	`9.74 ns` (➖ 1.03x slower)	`9.80 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`84.20 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.38 µs` (➖ 1.03x faster)	`2.25 µs` (➖ 1.01x faster)
`262144`	`10.96 µs` (➖ 1.03x slower)	`10.33 µs` (➖ 1.07x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.24 µs` (➖ 1.20x faster)	`8.19 µs` (➖ 1.27x faster)
`8192`	`1.28 µs` (➖ 1.02x faster)	`1.28 µs` (➖ 1.02x slower)
`262144`		`36.86 µs` (➖ 1.08x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`923.26 ns` (➖ 1.02x slower)
`vec_bytes`	`658.73 ns` (➖ 1.02x slower)
		`761.59 µs` (➖ 1.06x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`168.89 ns` (➖ 1.02x faster)
`262144`	`640.71 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`64.32 ms` (➖ 1.10x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.89 ns` (➖ 1.07x faster)	`8.76 ns` (➖ 1.07x faster)	`8.78 ns` (➖ 1.12x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`44.69 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.31 µs` (➖ 1.01x slower)	`2.28 µs` (➖ 1.00x faster)
`262144`	`7.95 µs` (➖ 1.01x faster)	`7.91 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.43 µs` (➖ 1.03x faster)	`7.46 µs` (➖ 1.03x faster)
`8192`	`924.59 ns` (➖ 1.03x faster)	`913.45 ns` (➖ 1.04x faster)
`262144`		`39.98 µs` (➖ 1.01x slower)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.15 µs` (➖ 1.01x slower)
`vec_bytes`	`766.08 ns` (➖ 1.00x faster)
		`1.88 ms` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`227.35 ns` (➖ 1.00x faster)
`262144`	`801.67 ns` (➖ 1.18x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`79.16 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.61 ns` (➖ 1.03x faster)	`10.54 ns` (➖ 1.03x faster)	`10.55 ns` (➖ 1.03x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`670.57 µs` (➖ 1.22x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.57 µs` (➖ 1.03x faster)	`2.54 µs` (➖ 1.00x faster)
`262144`	`8.64 µs` (➖ 1.04x faster)	`8.67 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`9.10 µs` (➖ 1.03x slower)	`9.21 µs` (➖ 1.01x faster)
`8192`	`1.64 µs` (➖ 1.01x faster)	`1.41 µs` (➖ 1.01x faster)
`262144`		`39.07 µs` (➖ 1.00x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.19 µs` (➖ 1.00x faster)
`vec_bytes`	`774.62 ns` (➖ 1.00x faster)
		`3.11 ms` (➖ 1.03x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`216.05 ns` (➖ 1.01x faster)
`262144`	`741.13 ns` (➖ 1.02x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`101.06 ms` (➖ 1.05x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.04 ns` (➖ 1.00x faster)	`11.08 ns` (➖ 1.08x slower)	`10.82 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`612.83 µs` (➖ 1.13x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`2.37 µs` (➖ 1.02x slower)	`2.47 µs` (➖ 1.09x slower)
`262144`	`7.86 µs` (➖ 1.02x slower)	`7.69 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`9.07 µs` (➖ 1.04x slower)	`8.66 µs` (➖ 1.01x faster)
`8192`	`1.52 µs` (➖ 1.02x faster)	`1.06 µs` (➖ 1.03x faster)
`262144`		`45.87 µs` (➖ 1.03x slower)

_{Reported by cargo ci bench-report --candidate run:39640691300 --config-file bench_report.toml.}

ludfjig · 2026-10-21T21:53:48Z

I wonder why we added sccache to begin with. Was it doing anything for C guest builds or anything?

I just think we didn't cache everything properly, so it kicked in where we cached the wrong thing or when cache was missing

ludfjig added 2 commits October 20, 2026 16:03

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 21, 2026

This comment has been minimized.

Sign in to view

ludfjig marked this pull request as ready for review October 21, 2026 01:37

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners October 21, 2026 01:37

Copilot AI balanced review requested due to automatic review settings October 21, 2026 01:37

Copilot started reviewing on behalf of ludfjig October 21, 2026 01:38

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread .github/workflows/PrimeCaches.yml Outdated

Comment thread .github/workflows/dep_fuzzing.yml

Comment thread Justfile Outdated

ludfjig force-pushed the ci-cache-priming branch from 59be1d4 to 13a312b Compare October 21, 2026 03:14

This comment has been minimized.

Sign in to view

ludfjig added 9 commits October 21, 2026 10:05

ludfjig force-pushed the ci-cache-priming branch from 13a312b to 3d16ce3 Compare October 21, 2026 17:05

danbugs approved these changes Oct 21, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

ludfjig merged commit da688e9 into hyperlight-dev:main Oct 21, 2026
160 of 182 checks passed

Conversation

ludfjig commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

syntactically commented Oct 21, 2026

Uh oh!

hyperlight-gh-bot Bot commented Oct 21, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

ludfjig commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ludfjig commented Oct 21, 2026 •

edited

Loading