You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
OnPair trainer should draw Rust's sample (StdRng + partial_shuffle) for byte-identical dictionaries #534
OnPairEncodingEncoder does not draw the same training sample as the Rust onpair crate, so it trains different dictionaries on identical data. Prime directive: behavioral parity with Rust.
Found by comparing a 100k-row URL column (https://example.com/products/<n>/reviews?page=<m>, 7 chunks of 16,384 rows) written by vortex-java and by Rust 0.88.0 (the release vortex-jni uses), both at cascade 3 with global dict off. Both pick vortex.onpair per chunk.
dict_bytes_len per chunk:
chunk
Rust
Java
0
8191
8429
1
8525
8368
2
8447
8387
3
8363
8452
4
8379
8473
5
8757
5670
6
1872
1877
Chunks 0-4 end up within 0.04% in total, but Java's chunk 5 is 18% smaller (131,904 B vs 161,507 B): its trainer built a 761-token dictionary and the cascade then dict-encoded the 467 distinct codes. That one chunk is 29.6 KB of the 31.6 KB (3.2%) by which Java's file is smaller. This is training luck, not a systematic advantage, and it could go the other way on other data.
Cause
Rust (onpair 0.2.1, training_order in encoding/trainer.rs): StdRng::seed_from_u64(42) (ChaCha12, rand 0.9.5) and order.partial_shuffle(&mut rng, k) with k = min(n, 2 * sample_fraction * n + 1024). Training consumes only the selected tail order[n-k..]; if that tail is shorter than the byte budget, k doubles and the shuffle restarts.
Java (OnPairEncodingEncoder.Encoder.train): SplittableRandom(42), a full Fisher-Yates shuffle, and training consumes order from the start.
Proposed fix
Port rand 0.9.5 partial_shuffle on top of SampleRng (already a bit-exact ChaCha12 StdRng and random_range for u32). It uses IncreasingUniform / calculate_bound_u32, which batches several swaps into one random chunk. Check that SampleRng.nextIntInclusive (written against rand 0.10) gives the same values as 0.9.5's random_range.
Reproduce the k computation, the tail consumption and the doubling retry loop in train.
Parity test: the URL column above, Rust as the oracle, comparing dict_bytes_len and encoded size per chunk (expect identical).
Repro
Scratch crate with vortex = "=0.88.0"; data dumped from the Java CascadeDepthGainIntegrationTest patterns (utf8-urls), seed 42. Not committed.
Activity