Repository navigation
perf(writer): align cascade stats with Rust's compressor (dense/run-aware counts, lazy ArrayAndStats) - #464
Merged
Merged
Conversation
This was referenced Oct 9, 2026
Merged
…ware counts, lazy ArrayAndStats) ## perf(writer): count integer distincts as Rust's compressor stats do Port vortex-compressor's integer stats loop (#10037 run-aware, #10041 dense counting): the min/max picks a dense `counts[value - min]` array for ranges under 256, or under 65536 and the row count; otherwise the hash map is probed once per run of equal values instead of once per row. Counts are identical, so every encoder verdict is unchanged. JavaVsJniWriteBenchmark.javaWriteCascading (10M OHLC rows, M-series): async-profiler, integer stats CPU: ~370 -> ~330 ms/op (~2% of a write) throughput, 5 forks: main 0.433 +- 0.014 -> 0.385 +- 0.017 ops/s gc.alloc.rate.norm: 6.89 GB -> 7.10 GB per op (+3%, widening to long[]) The throughput delta is inside this benchmark's fork-to-fork noise (main alone ranges 0.411-0.459 per fork; a 3-fork rerun of this commit gave 0.457-0.460 on its best fork). Small because the costliest integer column (volume, ~1.4M range, no runs) stays hashed and the dense path still costs ~2.7 ns/row. ## perf(writer): split bit-packing's min/max scan from its width histogram The per-row loop compared min/max through a signedness ternary, tested for negatives, and branched on zero width, all next to the histogram's scatter store. Min/max now run in their own branch-free loop (unsigned order via a sign-bit flip), negatives follow from the min, and width 0 falls out of numberOfLeadingZeros(0) == 64. Output is unchanged. JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks, on top of the previous commit: 0.385 +- 0.017 -> 0.422 +- 0.018 ops/s (main: 0.433 +- 0.014). gc.alloc.rate.norm unchanged (7.10 GB/op). Again within the benchmark's ~5% fork-to-fork noise. ## refactor(writer)!: compute cascade stats lazily through ArrayAndStats Mirror vortex-compressor's ArrayAndStats (stats/cache.rs): the cascade bundles the array with its stats, and the one scan (merged StatsOptions) runs on the first ArrayAndStats#stats() call, shared by every later candidate. EncodingEncoder#expectedRatio now takes (DType, ArrayAndStats), as Rust's expected_compression_ratio takes &ArrayAndStats. Groundwork rather than a speedup: every built-in stats consumer (Constant, Dict, RunEnd, Sparse) reads stats in the first sweep, so the scan still always runs. It is the hook for sharing more per-array stats (min/max for bit-packing and FoR) later. JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks: 0.442 +- 0.002 ops/s (previous commit 0.422 +- 0.018, main 0.433 +- 0.014); gc.alloc.rate.norm 7.10 GB/op, unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
dfa1
force-pushed
the
perf/compressor-stats-parity
branch
from
October 9, 2026 07:36
3c7d646 to
838082d
Compare
This was referenced Oct 9, 2026
dfa1
added a commit
that referenced
this pull request
Oct 9, 2026
Re-measured on main after #464-#469 (Apple M5, Zulu JDK 25.0.2, vortex-jni 0.86.1): reads 3 forks, the cascading write 5 forks. - README: an "at a glance" table (encodings, read, write, size vs vortex-jni) linking to the benchmark section. - explanation.md: new read, write and big-file tables. Reads are now 1.6-2.3x on numeric columns (vortex-jni got 3-6x faster since 0.79), ~4x on strings, 3.4x on the 3 GB scan. The write is on par (0.881 vs 0.848), with the caveats from #474/#477 and how #464-#469 got there. Drops the 2026-07 write table, whose "plain" row no longer exists (defaults cascade since #458). - withColumnEncoding: restricting candidates is now ~3.5x faster than the default cascade (24 -> 7 ms per 1M prices), not 6x -- the default path got cheaper. Updated in how-to, CHANGELOG and ColumnEncoding. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Aligns the cascade's stats with the current Rust
vortex-compressordesign (stats/integer.rs,stats/cache.rs), plus one bit-packing hot-loop cleanup. Output is byte-identical: every change preserves the counts each encoder decides on.Commits and numbers
JavaVsJniWriteBenchmark.javaWriteCascading(10M OHLC rows, cascading depth 3), JMH 5 forks × 5 iterations,-prof gc:b055154a)c0c89a61integer stats as Rust (#10037 run-aware, #10041 dense)f924b5b9bit-packing min/max split from histogram3c7d646elazyArrayAndStatsvortex-jni reference:
jniWrite0.848 ± 0.005 ops/s (3 forks).Read these honestly: fork-to-fork noise on this benchmark is about ±5% (main alone ranges 0.411–0.459; a 3-fork run of main an hour earlier gave 0.453 ± 0.005). Per async-profiler the integer stats change cuts ~370 → ~330 ms/op of CPU (~2% of a write). That's below the noise, so the per-commit throughput deltas are not attributable. The final state is level with main (0.442 vs 0.433), not a speedup.
Why it's small, and where the 2× gap to vortex-jni actually is
CPU per write (async-profiler, inclusive): Java ~1.9 s vs Rust ~1.0 s. Stats were ~0.6 s of Java's, but:
volume(I64, ~1.4M range, no runs) stays hashed, so neither shortcut applies;The bigger Java-only costs are elsewhere: sample estimation
measureBestChild~500 ms/op, bit-packing ~490 vs ~140 ms/op, Dict ~260 vs ~105 ms/op.API
Breaking:
EncodingEncoder#expectedRatio(DType, Object, ArrayStats)→expectedRatio(DType, ArrayAndStats), matching Rust'sexpected_compression_ratio(&ArrayAndStats). Lazy stats are neutral today (all four built-in consumers read stats in the first sweep). They're the hook for sharing more per-array stats later, e.g. min/max for bit-packing/FoR, which still rescan.Testing
./mvnw verify -pl writer -am: unit, checkstyle, javadoc./mvnw verify -pl integration -am: 364 tests, 0 failures (slow-tagged tests excluded locally)ArrayStatsTest#compute_everyIntegerCountingPath_matchesPerRowReference(dense-always, dense-narrow, hashed with runs, hashed wide, U64 straddling 2^63),LongIntMapTest#increment_byCount_addsAWholeRunInOneProbe,ArrayAndStatsTest🤖 Generated with Claude Code