Skip to content

Use zstd for multiprocess IPC compression when available - #676

Open
akx wants to merge 1 commit into
gaogaotiantian:masterfrom
akx:zstd-when-possible
Open

Use zstd for multiprocess IPC compression when available#676
akx wants to merge 1 commit into
gaogaotiantian:masterfrom
akx:zstd-when-possible

Conversation

@akx

@akx akx commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

A lot of time was being spent in zlib_compress; on modern Pythons, we can use zstd for free.

Refs #674 (and conflicts with it).

@akx
akx force-pushed the zstd-when-possible branch 4 times, most recently from f26648b to e3a4678 Compare March 31, 2026 09:49
@codecov

codecov Bot commented Mar 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (db53893) to head (b95b396).

Additional details and impacted files
@@             Coverage Diff             @@
##           master      #676      +/-   ##
===========================================
+ Coverage   99.95%   100.00%   +0.04%     
===========================================
  Files          20        21       +1     
  Lines        2473      2488      +15     
===========================================
+ Hits         2472      2488      +16     
+ Misses          1         0       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gaogaotiantian

Copy link
Copy Markdown
Owner

Do you have benchmark to support the idea?

@akx
akx force-pushed the zstd-when-possible branch from e3a4678 to 32837d2 Compare May 21, 2026 07:11
@akx

akx commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

@gaogaotiantian Sure. Looks to be 4% faster on a trivial small example, likely more on a real program:

$ hyperfine 'git checkout master && .venv/bin/viztracer foot.py' 'git checkout zstd-when-possible && .venv/bin/viztracer foot.py'
Benchmark 1: git checkout master && .venv/bin/viztracer foot.py
  Time (mean ± σ):      2.707 s ±  0.034 s    [User: 2.495 s, System: 0.201 s]
  Range (min … max):    2.664 s …  2.756 s    10 runs

Benchmark 2: git checkout zstd-when-possible && .venv/bin/viztracer foot.py
  Time (mean ± σ):      2.603 s ±  0.022 s    [User: 2.393 s, System: 0.201 s]
  Range (min … max):    2.565 s …  2.643 s    10 runs

Summary
  git checkout zstd-when-possible && .venv/bin/viztracer foot.py ran
    1.04 ± 0.02 times faster than git checkout master && .venv/bin/viztracer foot.py

@akx
akx force-pushed the zstd-when-possible branch from 32837d2 to b95b396 Compare May 21, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants