Python: Limit Expanded Bytes When Reading gzip
A tiny compressed file can contain much more data than its size suggests. For a small import, check the number of expanded bytes before decoding text or handing the payload to another parser. This example also tests the exact limit, concatenated members, and a damaged trailer.
Prepare a scratch directory
Tested on Linux with Python 3.12.14 and zlib 1.3.2. Use Python 3.12 or newer, Bash, a text editor, and mktemp, rm, and rmdir. No third-party packages are needed. Keep the same terminal open and stop if setup fails:
start_directory="$PWD"
work_directory="$(mktemp -d)" || exit 1
cd -- "$work_directory" || exit 1Read one byte past the allowance
Save as bounded-gzip-reader.py. All compressed fixtures stay in memory and contain only a few dozen bytes of original data:
import gzip
import io
import zlib
class ExpandedSizeError(ValueError):
pass
def read_small_gzip(source, limit):
if type(limit) is not int or not 0 <= limit <= 1024 * 1024:
raise ValueError("limit must be an integer from 0 to 1048576")
with gzip.GzipFile(fileobj=source, mode="rb") as reader:
payload = reader.read(limit + 1)
if len(payload) > limit:
raise ExpandedSizeError("expanded data exceeds limit")
return payload
def pack(payload):
return gzip.compress(payload, mtime=0)
fixtures = [
("below", pack(b"hello")),
("exact", pack(b"12345678")),
("over", pack(b"123456789")),
("two members", pack(b"12345") + pack(b"6789")),
("truncated", pack(b"hello")[:-4]),
]
for label, compressed in fixtures:
with io.BytesIO(compressed) as source:
try:
payload = read_small_gzip(source, limit=8)
except ExpandedSizeError:
print(f"{label}: rejected (expanded size)")
except (gzip.BadGzipFile, EOFError, zlib.error):
print(f"{label}: rejected (invalid gzip)")
else:
print(f"{label}: accepted {len(payload)} bytes")Run:
python3 bounded-gzip-reader.pyObserved output:
below: accepted 5 bytes
exact: accepted 8 bytes
over: rejected (expanded size)
two members: rejected (expanded size)
truncated: rejected (invalid gzip)Understand the extra byte
Reading only eight bytes cannot distinguish an eight-byte payload from a longer one. Asking for nine gives the function enough information to reject the larger input. The limit applies to the combined expanded stream, so splitting data into two gzip members does not reset the allowance.
For an accepted payload, the read reaches the end of the gzip stream. This matters because trailer errors can appear only after the data itself has been returned by earlier reads. Here a small truncated fixture is rejected rather than treated as a successful import.
The return value is bytes. Apply payload.decode("utf-8") afterward if the format requires UTF-8, and handle UnicodeDecodeError. An eight-byte allowance is not an eight-character allowance.
What this limit does and does not cover
The returned data is bounded, but this is not a complete resource sandbox. GzipFile has internal buffers; the limit is not an exact process-memory ceiling. It also does not impose a compressed-input limit or a time limit. A stream containing many empty members, or a slow input source, can consume work without producing many output bytes.
Use this pattern for small payloads with a trusted maximum allowance. At a service boundary, also bound compressed input and execution time. Do not first call gzip.decompress() on the entire untrusted payload and only then check its length.
The helper leaves the supplied source open; the caller owns it. The example closes each BytesIO explicitly. For a local file, supply a binary file opened in its own with block. See the Python gzip reference for stream ownership and invalid-input exceptions.
Clean up
rm -- bounded-gzip-reader.py
cd -- "$start_directory" || exit 1
rmdir -- "$work_directory"
unset start_directory work_directoryIf the directory is not empty, inspect its contents before removing anything else.
Acknowledgements
This post was created with assistance from AI.