Python Basics: Decode UTF-8 Across Chunk Boundaries

A network read or compressed-data chunk can end halfway through a character. Decoding each chunk separately fails even when the complete byte sequence is valid. Keep one decoder for the entire stream.

Prepare a scratch directory

Tested on Linux with CPython 3.12.14. You need Bash, Python, a text editor, and the standard mktemp, rm, and rmdir utilities. No external packages are required. Run without -O so assertions remain enabled.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as incremental-decoding-demo.py:

import codecs

chunks = [b"caf\xc3", b"\xa9 \xe2\x82", b"\xac"]
decoder = codecs.getincrementaldecoder("utf-8")(errors="strict")
parts = []
for chunk in chunks:
    part = decoder.decode(chunk, final=False)
    parts.append(part)
    print("Chunk text:", repr(part))
parts.append(decoder.decode(b"", final=True))
text = "".join(parts)
assert text == "café €"
print("Complete:", text)

truncated = codecs.getincrementaldecoder("utf-8")()
assert truncated.decode(b"\xe2\x82") == ""
try:
    truncated.decode(b"", final=True)
except UnicodeDecodeError:
    print("Final flush: truncated character rejected")
else:
    raise AssertionError("expected truncated UTF-8 to fail")

Run it:

python3 -B incremental-decoding-demo.py

Observed output:

Chunk text: 'caf'
Chunk text: 'é '
Chunk text: '€'
Complete: café €
Final flush: truncated character rejected

Read the result

The first call returns the complete ASCII prefix and retains the first byte of é. The next call finishes that character but retains the unfinished euro sign. Chunk boundaries do not need to match character boundaries. The final empty decode flushes pending state and detects the deliberately incomplete second stream.

Apply it carefully

Use a fresh decoder for each independent stream. Do not silently use errors="ignore" for structured records: dropping bytes can change their meaning. Decoding does not identify record boundaries; buffer or parse lines separately after decoding. This fixture collects all text for inspection; a large stream should consume decoded pieces incrementally.

The official codecs reference documents the APIs used here.

Clean up

Remove the saved source, return to the starting directory, and remove the empty practice directory:

rm -- incremental-decoding-demo.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect its remaining contents before removing anything else.

Acknowledgements

This post was created with assistance from AI.