Skip to content
Python: Limit Expanded Bytes When Reading gzip

Python: Limit Expanded Bytes When Reading gzip

November 10, 2026

A tiny compressed file can contain much more data than its size suggests. For a small import, check the number of expanded bytes before decoding text or handing the payload to another parser. This example also tests the exact limit, concatenated members, and a damaged trailer.

Prepare a scratch directory

Tested on Linux with Python 3.12.14 and zlib 1.3.2. Use Python 3.12 or newer, Bash, a text editor, and mktemp, rm, and rmdir. No third-party packages are needed. Keep the same terminal open and stop if setup fails:

start_directory="$PWD"
work_directory="$(mktemp -d)" || exit 1
cd -- "$work_directory" || exit 1

Read one byte past the allowance

Save as bounded-gzip-reader.py. All compressed fixtures stay in memory and contain only a few dozen bytes of original data:

import gzip
import io
import zlib


class ExpandedSizeError(ValueError):
    pass


def read_small_gzip(source, limit):
    if type(limit) is not int or not 0 <= limit <= 1024 * 1024:
        raise ValueError("limit must be an integer from 0 to 1048576")
    with gzip.GzipFile(fileobj=source, mode="rb") as reader:
        payload = reader.read(limit + 1)
    if len(payload) > limit:
        raise ExpandedSizeError("expanded data exceeds limit")
    return payload


def pack(payload):
    return gzip.compress(payload, mtime=0)


fixtures = [
    ("below", pack(b"hello")),
    ("exact", pack(b"12345678")),
    ("over", pack(b"123456789")),
    ("two members", pack(b"12345") + pack(b"6789")),
    ("truncated", pack(b"hello")[:-4]),
]
for label, compressed in fixtures:
    with io.BytesIO(compressed) as source:
        try:
            payload = read_small_gzip(source, limit=8)
        except ExpandedSizeError:
            print(f"{label}: rejected (expanded size)")
        except (gzip.BadGzipFile, EOFError, zlib.error):
            print(f"{label}: rejected (invalid gzip)")
        else:
            print(f"{label}: accepted {len(payload)} bytes")

Run:

python3 bounded-gzip-reader.py

Observed output:

below: accepted 5 bytes
exact: accepted 8 bytes
over: rejected (expanded size)
two members: rejected (expanded size)
truncated: rejected (invalid gzip)

Understand the extra byte

Reading only eight bytes cannot distinguish an eight-byte payload from a longer one. Asking for nine gives the function enough information to reject the larger input. The limit applies to the combined expanded stream, so splitting data into two gzip members does not reset the allowance.

For an accepted payload, the read reaches the end of the gzip stream. This matters because trailer errors can appear only after the data itself has been returned by earlier reads. Here a small truncated fixture is rejected rather than treated as a successful import.

The return value is bytes. Apply payload.decode("utf-8") afterward if the format requires UTF-8, and handle UnicodeDecodeError. An eight-byte allowance is not an eight-character allowance.

What this limit does and does not cover

The returned data is bounded, but this is not a complete resource sandbox. GzipFile has internal buffers; the limit is not an exact process-memory ceiling. It also does not impose a compressed-input limit or a time limit. A stream containing many empty members, or a slow input source, can consume work without producing many output bytes.

Use this pattern for small payloads with a trusted maximum allowance. At a service boundary, also bound compressed input and execution time. Do not first call gzip.decompress() on the entire untrusted payload and only then check its length.

The helper leaves the supplied source open; the caller owns it. The example closes each BytesIO explicitly. For a local file, supply a binary file opened in its own with block. See the Python gzip reference for stream ownership and invalid-input exceptions.

Clean up

rm -- bounded-gzip-reader.py
cd -- "$start_directory" || exit 1
rmdir -- "$work_directory"
unset start_directory work_directory

If the directory is not empty, inspect its contents before removing anything else.

Acknowledgements

This post was created with assistance from AI.

Last updated on