pickle vs json: Round-trip Lab

Intro — what this post promises

Local object cache / same-host IPC: is pickle faster and smaller than compact json for modest dict/list trees? This lab measures dumps / loads / round-trip ops/s and byte size on Linux localhost.

Security warning: pickle.loads on untrusted bytes is a remote-code-execution hazard. Use pickle only for trusted local data you wrote yourself.

Related links:

• 01shoppercove.comjson vs orjson vs msgpack localhost lab
• 02shoppercove.comjson dumps compact vs indent localhost lab
• 03shoppercove.comcsv reader vs split localhost lab
• 04shoppercove.comcopy vs deepcopy localhost lab
• 05shoppercove.comhashlib md5 vs blake2b localhost lab
• 06shoppercove.combytes vs bytearray localhost lab
• 07shoppercove.comshutil copyfile vs manual localhost lab
• 08shoppercove.comperf_counter vs time localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5; protocols [4, 5] (HIGHEST=5). JSON uses compact separators. Affiliates: 0. Not a rematch of orjson/msgpack (lab 36) or indent formatting (lab 75).

Verdict up front (small, 100 rows): pickle p5 dumps ~25433/s vs json ~7640 (~3.33×); round-trip ~2.64×; size 7,342 B vs 11,368 B (pickle/json ~0.65×). Prefer pickle for trusted local caches; JSON for anything crossing a trust boundary.

Arms

Payloads: tiny (10), small (100), medium (1000) nested row dicts.

Lab topology

trials: tiny 5k / small 1k / medium 100
metric: p50 ops/s + out bytes

Script: lab-evidence/79-pickle-vs-json-roundtrip/results/run_lab.py.

Lead table — small payload (p50)

Medium scale & size

pickle÷json dumps medium ~2.42×; size ratio ~0.65×. Protocol 5 ≈ 4 here (dumps p5÷p4 small ~0.99×).

Reading it

• Pickle wins trusted local encode/decode on these JSON-friendly dict trees — binary, no UTF-8 number formatting.
• JSON stays the interchange format for APIs, logs, and anything untrusted readers touch.
• Size — pickle was ~35% smaller than compact JSON on small/medium.
• Never pickle.loads from the network, a user upload, or a shared queue without a hard trust boundary.

Trust boundary checklist

1. Did your process write these bytes in this deployment? → pickle may be OK for a local cache file under your lock.
2. Did they cross a network, user upload, or multi-tenant queue? → JSON/msgpack/protobuf — never pickle.
3. Do non-Python readers need the blob? → JSON (or a schema’d binary format).

Round-trip speed does not override that checklist.

Protocol 4 vs 5

On these pure-dict payloads, protocol 5 was essentially tied with 4. Prefer protocol=5 (or HIGHEST_PROTOCOL) for new local caches anyway — newer opcodes help other object graphs (e.g. large binary out-of-band in some cases). Re-bench if your payload is numpy-heavy or self-referential.

Pitfalls

1. Unpickling untrusted input — classic RCE.
2. Using pickle across Python major versions / non-Python clients — fragile.
3. Pretty JSON as a cache format — waste (see lab 75).
4. Assuming protocol 5 always faster — measure; often a wash for pure-Python dicts.

When to pick what

Reproduce

python3 lab-evidence/79-pickle-vs-json-roundtrip/results/run_lab.py

Evidence: /workspace/lab-evidence/79-pickle-vs-json-roundtrip/results/.

Closing

Pickle for trusted local speed/size; JSON for boundaries. On this box small pickle p5 dumps beat compact JSON by ~3.3× with ~0.65× the bytes — and still must never load strangers’ pickles.