Python Basics: Check XZ Stream Completion and Trailing Bytes

An XZ payload can be followed by unrelated bytes. The incremental LZMA decoder exposes both stream completion and trailing data, making it possible to reject a payload that is not exactly one stream.

Prepare a scratch directory

Tested on Linux with CPython 3.12.14. You need Bash, a text editor, the named runtime, and standard mktemp, rm, and rmdir utilities. Run Python without -O so assertions remain enabled. The examples use only controlled local data; generated fixture directories are cleaned automatically.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as demonstrate-lzma.py:

import lzma

payload = b"alpha\nbeta\n"
packed = lzma.compress(payload, format=lzma.FORMAT_XZ)
decoder = lzma.LZMADecompressor(format=lzma.FORMAT_XZ)
restored = decoder.decompress(packed + b"TAIL")
assert restored == payload
assert decoder.eof and decoder.unused_data == b"TAIL"
print("Rows:", len(restored.splitlines()))
print("Stream complete:", decoder.eof)
print("Trailing bytes:", decoder.unused_data.decode("ascii"))

truncated = lzma.LZMADecompressor(format=lzma.FORMAT_XZ)
truncated.decompress(packed[:-1])
assert not truncated.eof
print("Truncated stream complete:", truncated.eof)
assert lzma.decompress(packed) == payload
print("Whole stream round trip: identical")

Run it:

python3 -B demonstrate-lzma.py

Observed output:

Rows: 2
Stream complete: True
Trailing bytes: TAIL
Truncated stream complete: False
Whole stream round trip: identical

Read the result

eof becomes true only after the decoder reaches the stream end. unused_data holds the tail after that end. Removing the final byte leaves eof false even if some or all application bytes have already been produced.

Apply it carefully

Do not treat output alone as proof that a stream was complete. For a single-stream protocol require eof and reject unexpected unused_data. For untrusted data, use decoder memory limits and an overall output limit; the tiny one-shot allocations here are not a general resource policy.

The official reference describes the behavior and options. These checks cover the fixture above rather than every input or platform.

Clean up

Remove the saved source, return to the starting directory, then remove the empty practice directory:

rm -- demonstrate-lzma.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect its remaining contents rather than deleting unknown files.

Acknowledgements

This post was created with assistance from AI.