groupby vs Manual Group: Localhost Lab

Intro — what this post promises

Group a pre-sorted list of (key, value) records: itertools.groupby vs a manual run-length scan vs defaultdict(list). This lab reports records/s on Linux localhost, and shows what happens if you feed unsorted data to groupby.

It is not Counter tallying (lab 64) or chain/flatten (lab 67), and not the itertools micro-loop bake-off (lab 55).

Related links:

• 01shoppercove.comcounter vs dict tally localhost lab
• 02shoppercove.comitertools chain vs flatten localhost lab
• 03shoppercove.comitertools vs python loops localhost lab
• 04shoppercove.comsetdefault vs defaultdict localhost lab
• 05shoppercove.comfnmatch vs re localhost lab
• 06shoppercove.comfutures as completed vs wait localhost lab
• 07shoppercove.comhmac compare digest localhost lab
• 08shoppercove.comargparse vs sys argv localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Affiliates: 0. No Docker. Input sorted unless noted.

Verdict up front (50 000 records / 200 keys): groupby consume ~24.21 Mrec/s; manual sorted runs ~18.09; defaultdict ~19.08; groupby→dict lists ~17.24. On shuffled input, naive d[k]=list(grp) lost 49800 / 50000 values.

Arms

Lab topology

n×keys: 10k/100, 50k/200, 200k/500 · sorted · 7 rounds · p50
unsorted demo: shuffle 50k, compare naive groupby vs defaultdict
metric: records/s = n / p50_seconds

Script: lab-evidence/98-groupby-vs-manual/results/run_lab.py.

Lead table — 50 000 records (p50 Mrec/s)

At this scale, consume is fastest because it never allocates value lists. Once you materialize, manual runs, groupby→dict, and defaultdict sit in a tight band — list appends dominate.

Scale sketch (groupby → dict vs defaultdict)

Throughput falls as group and list sizes grow — expected allocator pressure, not a mystery.

Unsorted footgun

Naive assign-per-run on shuffled data kept only 200 values (lost 49800). Merging runs or using defaultdict kept all 50000. groupby requires sorted (or already-grouped) input.

Sort tax reminder

If your pipeline is unsorted, add sorted(..., key=) cost before celebrating groupby. For one-shot grouping of random keys, defaultdict often wins end-to-end even when per-record rates look close on a pre-sorted fixture. Manual run-length scans assume contiguity the same way groupby does.

Reading it

• Prefer groupby when data is already sorted and you stream group iterators.
• defaultdict(list) wins when order is arbitrary or you do not want a sort tax.
• Materializing lists dominates — consume iterators when you can.
• Manual run-length scan ≈ groupby→dict when both build lists.
• setdefault trails defaultdict on this box for the same list-building work.

Pitfalls

• Using groupby on unsorted data (silent wrong merges / lost values).
• Assuming groupby output keys are unique without sorting.
• Sorting a huge list just to use groupby when defaultdict would do.
• Confusing grouping with Counter (counts only — lab 64).
• Measuring only consume throughput then shipping code that builds full dicts.

Reproduce

python3 lab-evidence/98-groupby-vs-manual/results/run_lab.py

Evidence: summary.json, summary.txt.

Limits

One Linux box. Synthetic int values. Sort cost not included in sorted-arm timings (fixture pre-sorted). Warmup discarded; p50 of seven rounds.

Takeaway

On sorted 50 k records, groupby consume ~24.21 Mrec/s led; building dicts sat ~17.24–19.08 Mrec/s. Sort first or use defaultdict — unsorted naive groupby dropped 49800 values here.