Queue vs deque Threaded Handoff: Localhost Lab

Intro — what this post promises

Measure producer→consumer handoff across threads: queue.Queue, queue.SimpleQueue, and a collections.deque + Lock/Condition hand-roll. This lab reports messages/s on Linux localhost.

It is not the single-threaded deque-vs-list queue bake-off (lab 54) — that was same-thread push/pop. Here the cost is cross-thread synchronization.

Related links:

• 01shoppercove.comdeque vs list queue localhost lab
• 02shoppercove.comgroupby vs manual localhost lab
• 03shoppercove.commethodcaller vs getattr localhost lab
• 04shoppercove.comhmac compare digest localhost lab
• 05shoppercove.comargparse vs sys argv localhost lab
• 06shoppercove.comshelve vs pickle dict localhost lab
• 07shoppercove.comsubprocess run vs popen localhost lab
• 08shoppercove.comfutures as completed vs wait localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Affiliates: 0. No Docker. One producer + one consumer; sentinel None.

Verdict up front (200 000 msgs): SimpleQueue ~14.13 Mmsgs/s; deque+Lock batch64 ~3.09; deque notify-each ~1.58; Queue unbounded ~0.97; Queue maxsize=1024 ~0.88. Use Queue when you need blocking/task_done/join; SimpleQueue or a careful deque+Lock when both ends are yours and you want speed.

Arms

Lab topology

n in {50k, 200k, 500k} ints · 1 producer · 1 consumer · 7 rounds · p50
metric: msgs/s = n / p50_s

Script: lab-evidence/100-queue-vs-deque-handoff/results/run_lab.py.

Lead table — 200 000 messages (p50 Mmsgs/s)

SimpleQueue leads by a wide margin on this box. Classic Queue pays for a richer API (mutex + conditions + unfinished-task tracking). Bounded maxsize adds a little more contention under a tight producer.

Scale sketch

Ordering holds across sizes — good sign the ranking is not a one-shot noise spike.

API vs speed

• Queue: get/put timeouts, task_done/join, maxsize backpressure — pick when workers need join semantics.
• SimpleQueue: fast FIFO for “just hand me items”; no unfinished-task counter.
• deque+Lock: only when you control both ends and accept Condition bugs if you get wait/notify wrong. Batching notifies helps; notify-every-put approaches Queue cost.

Backpressure note

Queue(maxsize=1024) is slightly slower here because the producer sometimes blocks when the buffer fills. That is the point of bounded queues in production: protect memory at the cost of a few messages/s. Unbounded Queue and SimpleQueue can grow without limit under a slow consumer — fine for labs, risky for long-running services.

Sentinel pattern

Both ends agree on a poison-pill (None here). Without it, get() waits forever after the last item. For typed payloads, use a dedicated sentinel object, not a value that might appear in data.

Reading it

• Prefer Queue for worker pools that join() on completion.
• Prefer SimpleQueue for high-rate handoff without join bookkeeping.
• Hand-rolled deque+Lock can beat classic Queue but rarely beats SimpleQueue here — and is easier to get wrong.
• Lab 54’s single-thread deque speed does not transfer to threaded handoff.

Pitfalls

• Using deque without a lock across threads (data races).
• Forgetting a sentinel / poison pill and hanging forever.
• Measuring only unbounded Queue then shipping bounded workers.
• Confusing this with single-thread deque benches (lab 54).

Reproduce

python3 lab-evidence/100-queue-vs-deque-handoff/results/run_lab.py

Evidence: summary.json, summary.txt.

Limits

One Linux box. Int payloads only. Two threads. No multi-consumer fan-out. GIL still serializes Python bytecode — this measures sync overhead, not parallel CPU.

Takeaway

At 200 k messages, SimpleQueue ~14.13 Mmsgs/s led; deque batch ~3.09; classic Queue ~0.97. Pick Queue for the API; pick SimpleQueue (or a careful deque) when handoff rate is the product metric.