Repository navigation
fix(writer): port Rust's scheme verdicts (taxi 60 → 42 MB, tip_amount fixed; +27% write) - #466
Merged
Merged
Conversation
This was referenced Oct 9, 2026
dfa1
force-pushed
the
perf/estimate-verdict-pass
branch
from
October 9, 2026 07:42
7394965 to
b572dbe
Compare
This was referenced Oct 9, 2026
Closed
… fixed; +27% write) ## fix(writer)!: port Rust's skip verdicts and choose_best_scheme ordering Selection now follows Rust's choose_best_scheme (compressor/select.rs) for full arrays and samples alike: a verdict pass over the array's own stats first, then only the deferred candidates are trial-encoded, on a sample built only if one needs it. Estimate becomes a sealed interface with Estimate.ratio(double), Rust's EstimateVerdict::Ratio, scored as `baseline / ratio` sample bytes. expectedRatio takes the EncodeContext, which gains sample()/withSampling() and finishedCascading(), and ArrayStats always carries integer min/max (unsigned for U64) with Rust's min_is_negative / max_minus_min / max_ilog2. Ported verdicts (vortex-btrblocks/src/schemes): - BitPacking: skip when min is negative - ZigZag: skip unless min is negative, or once cascading is finished - ALP, RLE: skip once cascading is finished - Sequence: skip on samples; on the real array a checked sequence scores len / 2 - Constant: never on a sample (Rust detects constants in the compressor itself, not on samples) The bit-packing skip fixes NYC taxi tip_amount: ALP's integers hold rare negatives that 1024-row samples seldom contain, so bit-packing measured narrow, then hit a negative on the full chunk and packed it at 64 bits. Taxi 2024-01: 59,912,206 -> 43,784,526 bytes (vortex-jni 44,463,892) tip_amount 20,187,604 -> 4,465,764 (jni 4,620,456) total_amount 6,322,258 -> 5,914,882 (jni 5,625,640) OHLC 10M rows: unchanged at 58,012,218 bytes. JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks: 0.599 +- 0.010 -> 0.608 +- 0.018 ops/s (per fork 0.602 0.622 0.597 0.642 0.576) gc.alloc.rate.norm 3.64 -> 3.83 GB/op ## perf(writer): estimate FoR and integer Dict from stats, as Rust does Port Rust's two stats-only estimates, which compete as Estimate.ratio without encoding anything: - FoRScheme (schemes/integer/for_.rs): skip once cascading is finished, for a zero min, a constant, or when bit-packing would be as narrow (min >= 0); else type width / bits of (max - min). - IntDictScheme (builtins/dict/integer.rs): skip above value_count / 2 distinct; else raw bits over values at full width plus codes, bit-packed or run-length encoded (32 bits per run) if smaller. FloatDictScheme keeps deferring to the sample. Taxi 2024-01: 43,784,526 -> 42,331,782 bytes (vortex-jni 44,463,892) fare_amount 4,566,764 -> 3,525,076 (jni 4,295,432) total_amount 5,914,882 -> 5,620,242 (jni 5,625,640) mta_tax 335,791 -> 213,780 (jni 181,664) tip_amount 4,465,764 -> 4,608,064 (jni 4,620,456) OHLC 10M rows: 58,012,218 -> 58,247,834 bytes (vortex-jni 61,681,692). JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks: 0.608 +- 0.018 -> 0.744 +- 0.021 ops/s (per fork 0.766 0.786 0.725 0.722 0.721) gc.alloc.rate.norm 3.83 -> 2.53 GB/op ## fix(writer): sparse-encode around the most frequent value, as Rust does Port Rust's SparseScheme (schemes/integer/sparse.rs): the fill is the most frequent value, whatever it is, not a hardcoded 0, and the verdict is stats-only: skip a constant or a top value under 90%, else estimate value_count / (value_count - top). Floats skip, as Rust's float sparse scheme only takes null-dominated arrays and validity never reaches a primitive here (vortex.masked splits it off). The reader already decoded any fill value. Taxi 2024-01: 42,331,782 -> 42,323,238 bytes (vortex-jni 44,463,892) tpep_dropoff_datetime 6,612,630 -> 6,455,454 (jni 6,451,168) tolls_amount 506,534 -> 551,200 (jni 592,152) congestion_surcharge 514,604 -> 550,367 (jni 585,796) store_and_fwd_flag 67,227 -> 63,370 (jni 47,536) Airport_fee 403,102 -> 496,240 (jni 415,648) Airport_fee moves away: Java dict-encodes it with vortex.masked codes whose null slots hold a code, inflating the top value past 90%; Rust stores null as a dictionary entry, so its codes have no nulls and stay bit-packed. Same verdict, different input -- the fix is the dict null representation, not this rule. OHLC 10M rows: 58,247,834 -> 58,230,650 bytes (vortex-jni 61,681,692). JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks: 0.744 +- 0.021 -> 0.763 +- 0.023 ops/s (per fork 0.757 0.817 0.761 0.726 0.756) gc.alloc.rate.norm 2.53 -> 2.49 GB/op Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
dfa1
force-pushed
the
perf/port-rust-verdicts
branch
from
October 9, 2026 07:47
7cc6cc3 to
31df7d9
Compare
dfa1
added a commit
that referenced
this pull request
Oct 9, 2026
… Rust (#471) Rust's dict builder codes values first-seen and gives null the next code at its first appearance. The frequency remap existed for a Sparse that filled with 0; Sparse fills with the most frequent value since #466, so the remap only made the codes diverge. Null is now an ordinary entry in DictColumnState, so the buffered codes are the wire codes: drops the remap, the per-code counts, emitCodes and withNullSlot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #465 (which is stacked on #464).
Ports the remaining Rust scheme verdicts (
vortex-btrblocks/src/schemes/*expected_compression_ratio) and Rust'schoose_best_schemeordering. These rules encode a lot of hard-won tuning upstream; Java had been deferring to sample measurement where Rust skips or estimates, and samples can mislead.Commits and numbers
JavaVsJniWriteBenchmark.javaWriteCascading(10M OHLC rows), JMH 5 forks × 5 iterations,-prof gc. File sizes per column viaInspectorTree, vortex-jni writing the same input:c0d9c459skip verdicts +choose_best_schemef74304cdFoR / integer Dict estimates7cc6cc39Sparse around the most frequent valueCascading write vs vortex-jni: 1.42× → 1.11× slower. Taxi: Java went from 35% bigger than vortex-jni to 5% smaller.
tip_amount (4.4× → 1.00× vortex-jni)
Fixed by Rust's BitPacking verdict (skip when min < 0) in
c0d9c459. ALP's integers fortip_amounthold rare negatives that 1024-row samples seldom contain. Bit-packing measured narrow on the sample, then hithasNegativeon the full chunk and packed it at 64 bits (8 B/value). With the full array's stats checked first, FoR wins: 20,187,604 → 4,608,064 bytes (jni 4,620,456).What changed structurally
Estimate→ sealed interface withEstimate.ratio(double)(Rust'sEstimateVerdict::Ratio), scored asbaseline / ratiosample bytes, equivalent to Rust's higher-ratio-wins with a ratio > 1.expectedRatio(DType, ArrayAndStats, EncodeContext).EncodeContextgainssample()/withSampling()/finishedCascading(). The SPI is unreleased since perf(writer): align cascade stats with Rust's compressor (dense/run-aware counts, lazy ArrayAndStats) #464, so the CHANGELOG entry is amended rather than stacked.ArrayStatsalways carries integer min/max (unsigned for U64) with Rust'smin_is_negative,max_minus_min,max_ilog2.len / 2on a checked sequence), Constant (never on samples), FoR, integer Dict (estimate incl. RLE codes), float Dict threshold, Sparse (fill = most frequent value, ≥ 90%).Known divergence (not caused here, surfaced here)
Airport_fee0.97× → 1.19× vortex-jni in7cc6cc39. Java dict-encodes nullable F64 columns withvortex.maskedcodes whose null slots hold a code. That inflates the top value past Sparse's 90%. Rust stores null as a dictionary entry, so its codes have no nulls. Same verdict, different input. The fix is Rust's null-as-dict-entry for primitive dicts (Java already does it for strings).Testing
./mvnw verify -pl writer -am: 1998 tests, checkstyle, javadoc./mvnw verify -pl integration -am: 364 tests, 0 failures (incl. Rust reading Java's non-zero-fill Sparse)SchemeVerdictTest(each ported rule's edges, U64 and negative-max stats), Sparse non-zero/negative-fill round trips🤖 Generated with Claude Code