zstd vs gzip vs lz4: Compression Throughput Lab

Intro — what this post promises

“Just enable gzip” and “switch to zstd” are slide-deck sentences. Throughput and ratio depend on payload entropy and level, and pathological repeating text will flatter any codec.

This is a hands-on lab with measured numbers:

1. Compress + decompress 64 MiB buffers in-process (Python bindings).
2. Codecs: gzip (default 9 + level 6), zstd (default 3, level 1, level 19), lz4 frame default.
3. Four payloads: pathological repeat, realistic /usr text, 50/50 mixed, os.urandom.
4. Report ratio, compress MB/s, decompress MB/s (p50 wall).

Related links:

• 01shoppercove.comnginx gzip on vs off localhost lab
• 02shoppercove.commmap vs read / O_DIRECT localhost lab
• 03shoppercove.comsendfile vs userspace copy localhost lab
• 04shoppercove.comWhy your average latency graph is lying (p50 / p95 / p99)
• 05shoppercove.comstdout buffering line vs full localhost lab

Lab honesty (1 Oct 2026 IST): Shared Linux lab box (8 cores, kernel 6.12). Python 3.13.5; python-zstandard 0.23.0 (CLI zstd 1.5.7); lz4 4.4.0+dfsg (CLI 1.10.0); system gzip 1.13. No Docker. Affiliates: 0. In-memory microbench — not HTTP Content-Encoding, not a threaded CLI bake-off.

Verdict up front: on realistic text, zstd default beat gzip9 by ~13× compress MB/s at a better ratio (2.39 vs 2.02). lz4 was the compress-speed king (~421 MB/s) with a weaker ratio (1.58). urandom stayed at ratio ≈1.000 for everyone.

What we are (and are not) measuring

Defaults used:

• gzip.compress → level 9 (Python default); also level 6 as a common “fast gzip” knob.
• zstandard.ZstdCompressor() → level 3; plus 1 and 19 for contrast.
• lz4.frame.compress → library default frame format.

Related links:

• 01shoppercove.comnginx gzip on vs off localhost lab

Lab topology

64 MiB payloads in RAM → compress → decompress → assert round-trip
Payloads:
  text_rep   — short English line repeated (pathological)
  text_real  — concat unique text-ish files under /usr/share + Python stdlib
  mixed      — 50% text_real + 50% os.urandom
  random     — os.urandom 64 MiB
Repeats: 5 (zstd19: 2–3) after 1 warmup; MB/s from p50 wall

First draft used a “random” buffer built from repeating SHA-256 blocks — that still compressed ~80–100×. We reran with true os.urandom and a real multi-file corpus. Evidence notes call that out; quote the rerun.

Lead table — realistic text (text_real, 64 MiB)

zstd3 vs gzip9: ~12.8× compress throughput, better ratio. lz4 wins raw compress speed here but gives up ~0.8 ratio points vs zstd3. zstd19 buys the best ratio (2.77) and loses two orders of magnitude on compress MB/s — offline archival vibes, not request path.

Related links:

• 01shoppercove.comWhy your average latency graph is lying (p50 / p95 / p99)

Mixed (50% text + 50% urandom)

Half entropy halves the fairy tale. Ratios collapse toward ~1.2–1.4; zstd/lz4 still crush gzip on compress speed.

Incompressible floor — os.urandom

Nobody shrinks pure noise (compressed size ≈ raw; gzip slightly over 100% with framing). Speed here is “how fast can I give up” — zstd still scans fast; zstd19 does not.

Pathological text_rep (do not cite as typical)

Repeating one short line: gzip ~294×, lz4 ~243×, zstd ~10 800× at multi‑GB/s. Useful as an upper bound and a reminder that synthetic compressibility lies. Lead with text_real, not this arm.

How to read these numbers

• Pick the payload class first — logs/HTML ≠ encrypted blobs ≠ /dev/urandom.
• Default zstd was the balanced pick on realistic text (ratio + speed).
• lz4 when CPU is dear and ratio can slip.
• gzip9 remains the compatibility default; it was the slow compress path here.
• Level 19 is a different product: ratio hobby, not RPS.

Related links:

• 01shoppercove.comnginx gzip on vs off localhost lab
• 02shoppercove.comprocess vs thread pool GIL localhost lab
• 03shoppercove.comasyncio vs threads IO concurrency lab

Pitfalls we hit (or avoided)

1. Fake random from repeating digests — still highly compressible; fixed with os.urandom.
2. Quoting pathological repeat ratios as “our logs.”
3. Comparing CLI multi-thread zstd to single-buffer Python without labeling — this post is Python bindings, one buffer.
4. Ignoring decompress — lz4/zstd both decompress fast here; gzip decompress was OK but compress was the bottleneck.
5. Assuming level 19 is “free ratio” — 2.5 MB/s on text_real says otherwise.

Practical checklist

• Measure on a slice of production-like bytes, not yes \| head.
• Record ratio + compress MB/s + decompress MB/s at the level you will ship.
• Prefer zstd default unless you need gzip clients or lz4 latency.
• Keep level 19 / gzip9 for offline or compatibility lanes.
• If the payload is already encrypted or random, skip compress — ratio ≈1.

Related links:

• 01shoppercove.comopenssl TLS full vs resume localhost lab
• 02shoppercove.comnice / ionice CPU and disk priority lab
• 03shoppercove.comcontext-switch pipe ping-pong localhost lab

Methodology notes (reproducible)

All arms used in-process Python:

• gzip.compress / gzip.decompress
• zstandard.ZstdCompressor(level=…) / ZstdDecompressor
• lz4.frame.compress / lz4.frame.decompress

Wall time is time.perf_counter around the full buffer op (not streaming chunk APIs). MB/s = 64 MiB / p50_seconds. Ratio = raw_bytes / compressed_bytes. Round-trip equality was asserted every repeat.

text_real was built by walking /usr/share/doc, /usr/share, and /usr/lib/python3.13, skipping ELF/gzip, deduping by a content hash prefix, concatenating until ≥64 MiB, then truncating. That is still “lab box text,” not your production JSON — but it is far closer than a single repeated sentence.

If you re-run on NVMe with the zstd CLI -T0, expect higher absolute MB/s; keep the relative ordering as the portable lesson unless you re-measure.

Related links:

• 01shoppercove.comepoll vs select FD_SETSIZE localhost lab
• 02shoppercove.comulimit soft vs hard EMFILE lab

Verdict

On 64 MiB realistic /usr text, zstd level 3 delivered ratio 2.39 at ~227 MB/s compress and ~639 MB/s decompress — roughly 13× faster compress than gzip9 (2.02 / 17.8 / 169) with a better ratio. lz4 hit ~421 MB/s compress at ratio 1.58. urandom stayed at ≈1.000 ratio for all codecs. Choose the codec for the entropy you actually ship, and keep fairy-tale repeating buffers in the appendix.

Evidence path on the lab box: lab-evidence/33-zstd-vs-gzip-lz4/results/. Affiliates: 0.