sqlite3 vs shelve Local KV: Localhost Lab

Intro — what this post promises

Local key-value persistence with sqlite3 vs shelve. This lab reports insert/get ops/s and on-disk bytes on Linux localhost for 5000 JSON-like records.

Related links:

• 01shoppercove.comfractions vs float localhost lab
• 02shoppercove.commath fsum vs sum localhost lab
• 03shoppercove.comitertools batched vs chunk localhost lab
• 04shoppercove.comexitstack vs nested with localhost lab
• 05shoppercove.comxml etree vs json localhost lab
• 06shoppercove.comlogging formatter vs fstring localhost lab
• 07shoppercove.comcopy copy vs dict copy localhost lab
• 08shoppercove.comhtml parser vs regex localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Affiliates: 0. Differentiates from shelve-vs-pickle-dict (lab 94) and sqlite WAL modes (lab 34) — here the peer is sqlite table KV vs shelve, default sqlite journal.

Verdict up front: sqlite insert ~217559 ops/s vs shelve insert ~13249; get: sqlite ~148945, shelve ~169920. Files: sqlite 532480 B, shelve 606208 B.

Arms

Seven rounds, p50. Temp dir on local disk. Values equal across stores (equal=True).

Lab topology

n=5000 keys · 7 rounds · p50
metric: ops/s = n / p50_s

Script: lab-evidence/128-sqlite3-vs-shelve/results/run_lab.py.

Lead table (p50 ops/s)

Insert is the headline gap: sqlite led by about 16.4×. Gets were close; shelve edged slightly on this run.

Disk footprint

After 5000 keys: sqlite 532480 bytes vs shelve aggregate 606208 bytes. Shelve pays pickle + dbm overhead; sqlite stores compact JSON text in one file.

Reading it for SRE work

• Bulk load / agent checkpoint writes → sqlite3 (insert throughput).
• Occasional Python-object cache on disk → shelve can be fine if write volume is low.
• Need SQL, indexes, concurrent readers → sqlite wins on features, not just speed.
• Lab 94 compared shelve to an in-process pickle dict; lab 34 compared WAL — this post is cross-API KV.

Document which store owns the file path in the runbook so on-call does not “migrate to shelve” during an incident without measuring inserts.

Write amplification note

Shelve insert sat near ~13249 ops/s because each assignment pickles through dbm. Sqlite batched inserts in one transaction hit ~217559. If your workload is read-heavy and already shelve-shaped, the get numbers (~169920 vs ~148945) matter more than insert bragging rights.

API shape vs speed

Shelve returns native dict-like objects without a JSON decode step — that helps explain get parity (~169920 ops/s vs sqlite ~148945). Insert still paid pickle+dbm. If your values are already JSON strings for a wire format, sqlite avoids a double encode; if values are rich Python graphs, shelve avoids a schema. Measure the operation you actually ship, not only the friendlier API.

Pitfalls

• Comparing shelve without sync to sqlite with commit.
• Assuming shelve is multi-process safe (it is not).
• Ignoring JSON encode cost on the sqlite path (included here on purpose).
• Treating lab 34 WAL timings as interchangeable with this default-journal run.

Reproduce

python3 lab-evidence/128-sqlite3-vs-shelve/results/run_lab.py

Evidence: summary.json, summary.txt.

Limits

One Linux box, tempfile disk, single writer. Not networked DB, not SQLCipher.

Takeaway

For 5000 local KV writes, sqlite3 insert ~217559 ops/s crushed shelve ~13249; gets were similar. Prefer sqlite for write-heavy local state; keep shelve for small Python-native caches.