feat(pyamber): preserve minus-one shuffle hashes - #8254
Conversation
Automated Reviewer SuggestionsBased on the
|
Backport auto-label reportThis
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8254 +/- ##
============================================
- Coverage 93.93% 93.90% -0.04%
+ Complexity 4762 4761 -1
============================================
Files 1191 1191
Lines 48308 49051 +743
Branches 5377 5377
============================================
+ Hits 45380 46059 +679
- Misses 1494 1557 +63
- Partials 1434 1435 +1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 548 | 0.335 | 17,599/22,605/22,605 us | 🔴 +20.5% / 🔴 +45.1% |
| 🔴 | bs=100 sw=10 sl=64 | 1,216 | 0.742 | 80,009/110,340/110,340 us | 🔴 +20.3% / 🟢 +20.6% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,415 | 0.864 | 713,312/746,101/746,101 us | ⚪ within ±5% / 🟢 +35.8% |
Baseline details
Latest main 50321e4 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 548 tuples/sec | 626 tuples/sec | 787.98 tuples/sec | -12.5% | -30.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.335 MB/s | 0.382 MB/s | 0.481 MB/s | -12.3% | -30.3% |
| bs=10 sw=10 sl=64 | p50 | 17,599 us | 14,609 us | 12,593 us | +20.5% | +39.8% |
| bs=10 sw=10 sl=64 | p95 | 22,605 us | 22,524 us | 15,579 us | +0.4% | +45.1% |
| bs=10 sw=10 sl=64 | p99 | 22,605 us | 22,524 us | 18,786 us | +0.4% | +20.3% |
| bs=100 sw=10 sl=64 | throughput | 1,216 tuples/sec | 1,227 tuples/sec | 1,008 tuples/sec | -0.9% | +20.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.742 MB/s | 0.749 MB/s | 0.615 MB/s | -0.9% | +20.6% |
| bs=100 sw=10 sl=64 | p50 | 80,009 us | 80,311 us | 100,701 us | -0.4% | -20.5% |
| bs=100 sw=10 sl=64 | p95 | 110,340 us | 91,695 us | 107,244 us | +20.3% | +2.9% |
| bs=100 sw=10 sl=64 | p99 | 110,340 us | 91,695 us | 116,122 us | +20.3% | -5.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,415 tuples/sec | 1,428 tuples/sec | 1,042 tuples/sec | -0.9% | +35.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.864 MB/s | 0.872 MB/s | 0.636 MB/s | -0.9% | +35.8% |
| bs=1000 sw=10 sl=64 | p50 | 713,312 us | 701,051 us | 981,959 us | +1.7% | -27.4% |
| bs=1000 sw=10 sl=64 | p95 | 746,101 us | 735,365 us | 1,023,080 us | +1.5% | -27.1% |
| bs=1000 sw=10 sl=64 | p99 | 746,101 us | 735,365 us | 1,051,697 us | +1.5% | -29.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,364.88,200,128000,548,0.335,17599.13,22604.67,22604.67
1,100,10,64,20,1644.71,2000,1280000,1216,0.742,80009.35,110339.80,110339.80
2,1000,10,64,20,14133.68,20000,12800000,1415,0.864,713311.75,746101.03,746101.03|
I am converting this PR to a draft: please follow our PR template and mark it ready for review. |
|
The description now follows the current template, and this is ready for review. |
What changes were proposed in this PR?
Hash shuffle now uses the tuple's raw Java-compatible hash method directly, avoiding Python's reserved
-1to-2normalization before receiver selection.Any related issues, documentation, discussions?
Closes #8253
How was this PR tested?
Direct reproduction after the fix:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Codex