Repository navigation
perf(writer): ALP exponent search as Rust's alp crate (cascading write now faster than vortex-jni) - #469
Merged
Conversation
This was referenced Oct 9, 2026
dfa1
force-pushed
the
fix/varbin-dict-null-entry
branch
from
October 9, 2026 07:57
9e4927b to
809a294
Compare
…e now faster than vortex-jni) ## perf(writer): search ALP exponents as Rust's alp crate does Port find_best_exponents and encode from the `alp` crate 0.0.4, which vortex 0.86.1 pins: - search on SamplePlan::subsample: at most 64 values, 8 evenly spaced runs of 8 (Java sampled 512 strided values) - every f <= e, e descending from (0, 0); ties go to the smaller e - f - estimate_encoded_size_within: stop a candidate once its exceptions alone cost more than the best so far - fast_round, (x + 2^52 + 2^51) - 2^52 - 2^51 (f32: 2^23 + 2^22), which Java's strict IEEE arithmetic reproduces bit for bit, instead of Math.round; round-trip checks bitwise, as is_eq - exception slots filled with the first value that did encode, not 0, so the integer child keeps its frame of reference The search runs on every ALP trial and real encode, so its sample size was most of ALP's cost (Java ~184 ms vs Rust ~30 ms of CPU per OHLC write). Output (vortex-jni writing the same input): OHLC price columns ~5.99 MB -> ~6.19 MB each (jni ~6.91 MB: 0.87x -> 0.90x); every other OHLC column byte-identical; total 58,230,658 -> 59,036,738 (jni 61,681,692) Taxi 2024-01: 42,340,950 -> 42,338,902 bytes (jni 44,463,892) JavaVsJniWriteBenchmark.javaWriteCascading, 5 forks: 0.767 +- 0.017 -> 0.881 +- 0.008 ops/s (per fork 0.886 0.864 0.891 0.874 0.889) vortex-jni: 0.848 ops/s gc.alloc.rate.norm 2.50 -> 2.50 GB/op Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
dfa1
force-pushed
the
perf/alp-exponent-search
branch
from
October 9, 2026 08:01
a9aa931 to
e07364f
Compare
dfa1
added a commit
that referenced
this pull request
Oct 9, 2026
Re-measured on main after #464-#469 (Apple M5, Zulu JDK 25.0.2, vortex-jni 0.86.1): reads 3 forks, the cascading write 5 forks. - README: an "at a glance" table (encodings, read, write, size vs vortex-jni) linking to the benchmark section. - explanation.md: new read, write and big-file tables. Reads are now 1.6-2.3x on numeric columns (vortex-jni got 3-6x faster since 0.79), ~4x on strings, 3.4x on the 3 GB scan. The write is on par (0.881 vs 0.848), with the caveats from #474/#477 and how #464-#469 got there. Drops the 2026-07 write table, whose "plain" row no longer exists (defaults cascade since #458). - withColumnEncoding: restricting candidates is now ~3.5x faster than the default cascade (24 -> 7 ms per 1M prices), not 6x -- the default path got cheaper. Updated in how-to, CHANGELOG and ColumnEncoding. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #468 (← #467 ← #466 ← #465 ← #464).
After #468, Java was still ~10% slower than vortex-jni. A CPU profile put the largest gap in ALP: Java ~184 ms vs Rust ~30 ms per OHLC write. The exponent search was the cause. It runs on every ALP trial and real encode, and Java searched 512 values where Rust's
alpcrate (0.0.4, pinned by vortex 0.86.1) searches 64.What's ported (
spiraldb/alpv0.0.4,src/alp/mod.rs,src/sample.rs)SamplePlan::subsample: at most 64 values, 8 evenly spaced runs of 8f <= e,edescending, starting from(0, 0); ties go to the smallere - festimate_encoded_size_within: a candidate stops once its exceptions alone cost more than the best so farfast_round((x + 2^52 + 2^51) - (2^52 + 2^51), f322^23 + 2^22) instead ofMath.round; Java's strict IEEE reproduces it bit for bit. Round-trip checks are bitwise, asis_eq.fill_value), not 0, so the integer child keeps its frame-of-reference rangeThe removed Java comment justified 512 samples by "precision drift" risk at small samples. Rust accepts that risk with 64; per the prime directive we take Rust's choice.
Numbers
javaWriteCascading(5 forks)Every other OHLC column is byte-identical; the price columns move toward vortex-jni.
Testing
./mvnw test -pl writer -am: 1995 tests (incl. the ALP round-trip suite)./mvnw verify -pl integration -am: 365 tests, 0 failures🤖 Generated with Claude Code
https://claude.ai/code/session_015R8cf8YM8XFQqrkyhSAtYr