Pipe vs Tmpfile IPC: Localhost Throughput and RTT Lab

Intro — what this post promises

Need to move bytes between processes on one box? Two defaults fight for the job: pipes (os.pipe / shell |) and temporary files. Folklore says “files are fine on SSD.” This lab measures both shapes with the same payload sizes.

This is a hands-on lab with measured numbers:

1. Bulk throughput: 64 MiB through a pipe vs write+read of a tmpfile (with and without fsync).
2. Request/response RTT: duplex pipe (fork parent↔child) vs a file “mailbox” open/write/read loop.
3. How chunk size and fsync change the story.
4. When a tmpfile is still the right tool despite losing the race.

Related links:

• 01shoppercove.comUnix Domain Socket vs TCP localhost lab
• 02shoppercove.comHTTP Keep-Alive vs Connection: close lab
• 03shoppercove.comWhy your average latency graph is lying (p50 / p95 / p99)
• 04shoppercove.comHow to read server monitoring graphs

Lab honesty (30 Sep 2026 IST): Shared Linux lab box (8 cores, kernel 6.12). Python 3.13.5. Pipes via os.pipe + fork; tmpfiles under /workspace (overlay). No Docker. No GPU. Affiliates: 0.

Verdict up front: best pipe bulk was 5.32 GB/s (64 KiB chunks). Same 64 MiB via tmpfile buffered hit 1.51 GB/s (3.5×~~ slower); with fsync 0.57 GB/s (~~9×). RTT for 64 B: pipe p50 3.3 µs (~285k RPS) vs tmpfile 70 µs (~12k) vs tmpfile+fsync 289 µs (~2.5k).

What we compared

Related links:

• 01man7.orgman 2 pipe
• 02man7.orgman 2 fsync

Honesty on RTT: the tmpfile arm is a same-process mailbox (open/write/read). That is an optimistic lower bound for “leave a file for the next stage.” The pipe RTT is a true two-process duplex. If anything, real multi-process file polling would be worse for files.

Lab topology

Throughput: 64 MiB  →  pipe(chunk)  |  tmpfile write→read  |  tmpfile+fsync
RTT:        sizes 1 / 64 / 1K / 64K  →  pipe duplex  |  file mailbox ± fsync
Confirm:    128 MiB throughput pass (same order of magnitude)

Arm A — bulk throughput (64 MiB)

Pipe peaked at 64 KiB chunks. Tmpfile never caught pipe on this pass; fsync roughly halved to thirded buffered tmpfile GB/s. A 128 MiB confirm pass kept the same ranking (pipe ~3.7 GB/s at 64 KiB vs tmpfile ~1.4 / fsync ~0.54).

Arm B — request/response RTT

For chatty handoffs, pipe’s ~21× p50 advantage over buffered files (and ~87× over fsync) is the operational punchline. Durability is not free.

When tmpfiles still win the design

Pipes lose data if nobody is reading and the buffer fills; they do not survive process restart; they are awkward for random access and for “many consumers later.” Tmpfiles (or real paths) win when you need replay, crash durability, multi-reader, or a simple ops story (ls the artifact). This lab measures speed, not those product constraints.

UDS sits in between for same-host RPC — see the UDS vs TCP lab — while pipes stay the cheapest streaming primitive inside one machine.

Related links:

• 01shoppercove.comUnix Domain Socket vs TCP localhost lab

Pitfalls we hit (or avoided)

1. Comparing pipe RTT to an unfair file poller — we used a same-process mailbox and said so; real watchers cost more.
2. Forgetting fsync — buffered GB/s looks fine until durability is required.
3. Wrong chunk size — pipe at 1 MiB was slower than 64 KiB here; measure your chunk.
4. Calling this shared-memory or mmap — different tools; not in this package.
5. Assuming NVMe makes files “as fast as pipes” — not for RTT-shaped IPC on this box.

Practical checklist

• Streaming one producer → one consumer on one host: start with a pipe (or UDS if you need sockets).
• Need durability or multi-stage replay: tmpfile/object store — budget the fsync tax.
• Measure p50/p95 for chatty IPC; mean alone hides fsync spikes.
• Tune chunk size; do not assume bigger is always better.
• Do not confuse this with network RPC or cross-host queues.

Verdict

Pipes crushed tmpfiles for both bulk and RTT on this box: 5.32 GB/s vs 1.51 GB/s buffered (64 MiB / 64 KiB chunks), and 3.3 µs vs 70 µs p50 for 64 B RTT — 289 µs once fsync entered. Use files when you need persistence or ops visibility; use pipes when you need streaming speed between live processes.

Evidence path on the lab box: lab-evidence/19-pipe-vs-tmpfile/results/. Affiliates: 0.