Counter vs dict: Tally Lab

Intro — what this post promises

Counting tokens: is collections.Counter faster than a manual dict or defaultdict(int)? This lab tallies N=200 000 tokens on Linux localhost across low / mid / high cardinality, comparing Counter.update, Counter(...), loop c[t]+=1, d.get, try/except, and defaultdict.

Related links:

• 01shoppercove.comfrozenset vs set membership localhost lab
• 02shoppercove.combytes vs bytearray localhost lab
• 03shoppercove.comenum vs constants localhost lab
• 04shoppercove.comitemgetter vs lambda sort localhost lab
• 05shoppercove.comcontextlib vs try/finally localhost lab
• 06shoppercove.comitertools vs python loops localhost lab
• 07shoppercove.comlru_cache hit vs miss localhost lab
• 08shoppercove.comarray vs list ints localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. One op = one token consumed. Affiliates: 0. Counter still wins on API (most_common, arithmetic); this is the tally hot path.

Verdict up front (mid card, vocab=5 000): Counter(tokens) ~23.4M/s, update ~22.7M, dict.get ~13.7M (update ~1.66× dict.get). Loop c[t]+=1 lags update ~2.5×. Prefer Counter(iterable) / .update, not per-item Counter math in pure Python.

Arms

Cardinalities: low vocab=50, mid 5 000, high 80 000.

Lab topology

N = 200000 tokens/arm; repeats=9; p50 ops/s
plus 500× most_common(10) vs sorted(items)[:10]

Script: lab-evidence/64-counter-vs-dict-tally/results/run_lab.py.

Lead table — mid cardinality (p50)

Low vs high card (update / dict.get / ctor)

At low card, dict.get stayed competitive (update ~0.92×); ctor still led (~1.41×).

Top-k after the tally

most_common ~3.9× a full sort-slice on the same mid-card map — heap-sized top-k vs sorting everything.

Reading it

• Counter.update / constructor are C-accelerated — beating a Python for with c[t]+=1 by ~2.5× mid-card.
• defaultdict ≈ dict.get here (~1.13× mid) — pick for style, not a huge win.
• Avoid Counter as a slow dict — the loop arm is the trap.
• Use Counter when you want Counter features — most_common, multiset ops — and feed it with update/ctor.

Why update beats c[t] += 1

Counter.__setitem__ from a Python loop pays interpreter overhead on every token. Counter.update and the constructor walk the iterable in C and batch the hash-table work. That is why mid-card update landed near ~23M/s while the loop arm sat near ~9M/s — same abstract algorithm, different implementation layer. If you already have a Counter and a stream, call update; if you are building from one iterable, pass it to Counter(...) directly.

Pitfalls

1. for t in tokens: c[t]+=1 — looks idiomatic, loses the C path.
2. try/except for missing keys — depends on hit rate (won mid, lost low/high here).
3. Sorting all items for top-10 — most_common(k) exists.
4. Microbenching empty Counter creation — amortize over real N.

When to pick what

Reproduce

python3 lab-evidence/64-counter-vs-dict-tally/results/run_lab.py

Evidence: /workspace/lab-evidence/64-counter-vs-dict-tally/results/.

Closing

Feed Counter in bulk. On this box mid-card update hit ~23M/s vs dict.get ~14M (~1.66×) and loop-+= trailed update ~2.5×. Construct or update; don’t reimplement Counter one key at a time in Python.