Line Buffering vs Full Buffering: Why Is My Output Late?

Intro — what this post promises

Line buffering writes stdout after every newline, so a Python child piping 40 lines showed its first line after about 12 ms, while full (8 KiB block) buffering, the default when stdout is a pipe or file rather than a terminal, held everything until about 226 ms. Full buffering was about 4.9x faster on throughput (about 4.0M vs 0.82M lines/s on a pipe), so keep it for bulk output and switch on python -u or PYTHONUNBUFFERED=1 when something reads the pipe live.

This is a hands-on lab with measured numbers:

1. Throughput for 200k lines under full (8 KiB), line (buffering=1), and unbuffered (buffering=0) modes.
2. First-line visibility when a child prints 40 lines at 5 ms pace into a pipe.
3. File-redirect throughput (same modes).
4. Default print() on a pipe vs python -u vs PYTHONUNBUFFERED=1.

Related links:

• 01shoppercove.comPipe vs tmpfile IPC localhost lab
• 02shoppercove.comWhy your average latency graph is lying (p50 / p95 / p99)
• 03shoppercove.comNginx gzip on vs off localhost lab
• 04shoppercove.comHow to read server monitoring graphs

Lab honesty (30 Sep 2026 IST): Shared Linux lab box (8 cores, kernel 6.12). Python 3.13.5. Child stdout is a pipe or file (not a TTY — so Python’s default is block-buffered). No Docker. No GPU. No API keys. Affiliates: 0.

Verdict up front: full buffering hit ~4.0M lines/s on a pipe but first line stayed invisible until ~226 ms. Line / unbuffered showed first line at ~12 ms. Default pipe print() matched full; python -u and PYTHONUNBUFFERED=1 restored early visibility.

What we compared

Related links:

• 01docs.python.orgTextIOWrapper buffering — Python docs
• 02docs.python.orgpython -u / PYTHONUNBUFFERED

Lab topology

Parent reads child stdout (pipe) or redirects to a tempfile
Child: write N lines under full | line | none, then DONE + flush
Arm A: 200k lines throughput (pipe)
Arm B: 40 lines @ 5 ms — first / mid / done visibility
Arm C: 200k lines → file
Arm D: default print() vs python -u vs PYTHONUNBUFFERED=1

Arm A — pipe throughput (200k lines, p50)

Block buffering wins raw throughput. Paying a write(2) per line (or nearly) is expensive when you do not need live visibility.

Arm B — visibility latency (40 lines × 5 ms, p50)

Full buffering held everything until the final flush — first, mid, and DONE arrived together near ~226 ms. Line and unbuffered streamed: first line ~12 ms after start.

Arm C — file redirect (200k lines, p50)

Same story on a file sink: full buffering dominates; line and none collapse together when every line is a syscall.

Arm D — default print() vs -u / env

Default print() to a pipe behaved like full buffering. Both -u and PYTHONUNBUFFERED=1 restored early first-line visibility (~13–14 ms).

Related links:

• 01shoppercove.comPipe vs tmpfile IPC localhost lab

When full buffering still wins

• High-volume batch logs where you only care about the finished file/pipe contents.
• CI artifacts / redirects where progress is not consumed live.
• Throughput-sensitive emitters (Arm A/C: ~4–5×–7× faster than line mode here).

Use line buffering (or -u / explicit flush) when a human, supervisor, or sidecar reads the pipe live.

Pitfalls we hit (or avoided)

1. Assuming TTY rules apply to pipes — non-TTY stdout is block-buffered by default.
2. Blaming “hung” jobs when logs are just sitting in the 8 KiB buffer.
3. Line-buffering every hot path — Arm A/C show the throughput tax.
4. Forgetting the final flush — full mode only became visible after flush() / exit.
5. Mixing parent bufsize with child buffering — we set parent pipe reads unbuffered so visibility reflects the child.

Practical checklist

• Live progress on pipes: python -u, PYTHONUNBUFFERED=1, or explicit flush=True / line buffering.
• Bulk redirects: prefer default/full buffering; do not pay per-line syscalls.
• Supervisors/sidecar log shippers: test with a pipe, not only an interactive TTY.
• Report sink type (TTY/pipe/file) + mode with every lines/s claim.
• Pair with pipe-vs-tmpfile when the question is IPC shape, not print buffering.

Verdict

On non-TTY stdout, full (8 KiB) buffering delivered ~4.0M lines/s (pipe) / ~4.3M (file) but hid the first line until ~226 ms. Line / unbuffered showed first line at ~12 ms and paid ~5–8× throughput. Default pipe print() matched full; python -u and PYTHONUNBUFFERED=1 fixed visibility without rewriting the child.

Evidence path on the lab box: lab-evidence/25-stdout-buffering/results/. Affiliates: 0.

FAQ

What is line buffering?

Output is flushed to the OS at each newline. In Python that's buffering=1 on a text stream. It is the default for interactive terminals, not for pipes or files.

Why does my Python script's output appear all at once?

When stdout is a pipe or file, Python block-buffers it (8 KiB here), so lines wait until the buffer fills or the process exits. In this lab, default print() to a pipe behaved like full buffering.

How do I make Python output unbuffered?

Run python -u, set PYTHONUNBUFFERED=1, or pass flush=True to print. Both -u and the environment variable brought the first line back to about 13-14 ms.