Python Basics: Read Compressed Text and Concatenated bzip2 Streams

A bzip2 file can be processed as text, while the byte API can decode concatenated compressed streams. The incremental decoder stops after one stream, which matters when parsing a combined payload.

Prepare a scratch directory

Tested on Linux with CPython 3.12.14. You need Bash, a text editor, the named runtime, and standard mktemp, rm, and rmdir utilities. Run Python without -O so assertions remain enabled. The examples use only controlled local data; generated fixture directories are cleaned automatically.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as demonstrate-bz2.py:

import bz2
from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as directory:
    path = Path(directory) / "events.bz2"
    with bz2.open(path, "wt", encoding="utf-8") as target:
        target.write("open\nclose\n")
    with bz2.open(path, "rt", encoding="utf-8") as source:
        rows = [line.rstrip("\n") for line in source]
    assert rows == ["open", "close"]
    print("Text rows:", ", ".join(rows))

combined = bz2.compress(b"first|") + bz2.compress(b"second")
assert bz2.decompress(combined) == b"first|second"
decoder = bz2.BZ2Decompressor()
first = decoder.decompress(combined)
assert first == b"first|" and decoder.eof
assert bz2.decompress(decoder.unused_data) == b"second"
print("One-shot:", bz2.decompress(combined).decode("ascii"))
print("Incremental first stream:", first.decode("ascii"))
print("Remaining stream:", bz2.decompress(decoder.unused_data).decode("ascii"))

Run it:

python3 -B demonstrate-bz2.py

Observed output:

Text rows: open, close
One-shot: first|second
Incremental first stream: first|
Remaining stream: second

Read the result

Text mode adds UTF-8 decoding above decompression. The convenience decompress function processes both streams. BZ2Decompressor instead exposes bytes after its first stream through unused_data, allowing the caller to decide what follows.

Apply it carefully

These are tiny trusted fixtures. One-shot decompression can allocate a large result; compressed size alone is not a useful output limit. For larger or untrusted inputs, bound total output and account for every stream rather than assuming one decoder consumes everything.

The official reference describes the behavior and options. These checks cover the fixture above rather than every input or platform.

Clean up

Remove the saved source, return to the starting directory, then remove the empty practice directory:

rm -- demonstrate-bz2.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect its remaining contents rather than deleting unknown files.

Acknowledgements

This post was created with assistance from AI.