Process only the bytes returned by readinto
readinto() fills a buffer you provide and returns the number of bytes read. A short read leaves the rest of that buffer unchanged. Processing its entire capacity can append old bytes to the new data.
Set up
Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as valid_buffer_prefix.py:
from io import BytesIO
payload = b"ABCDEFGHijk"
buffer = bytearray(8)
chunks = []
with BytesIO(payload) as source, memoryview(buffer) as view:
while True:
count = source.readinto(buffer)
if count == 0:
break
valid = bytes(view[:count])
chunks.append(valid)
print("Read:", count, "buffer:", bytes(buffer), "valid:", valid)
restored = b"".join(chunks)
assert chunks == [b"ABCDEFGH", b"ijk"]
assert restored == payload
assert bytes(buffer) == b"ijkDEFGH"
print("Restored:", restored)Run:
python3 -B valid_buffer_prefix.pyObserved output:
Read: 8 buffer: b'ABCDEFGH' valid: b'ABCDEFGH'
Read: 3 buffer: b'ijkDEFGH' valid: b'ijk'
Restored: b'ABCDEFGHijk'Separate capacity from valid length
The buffer keeps its eight-byte capacity throughout the loop. On the second read, only its first three bytes are replaced. The returned count identifies the valid prefix; clearing the tail is unnecessary when every consumer respects that count.
bytes(view[:count]) copies the prefix before the buffer is reused. Saving the view itself would keep a reference to mutable storage, so earlier chunks could change on the next read. Immediate consumers can use a view without retaining it, provided they finish before the buffer changes.
Keep stream handling explicit
The example uses a blocking in-memory binary stream and a nonempty buffer. Zero signals EOF here. A short positive read must still be processed; do not discard it or assume every call fills the buffer. Nonblocking streams need separate would-block handling, which can involve None or BlockingIOError depending on the interface.
Collecting all chunks retains the full payload in memory. A streaming application can process each valid prefix immediately instead. Byte chunks also need not align with text characters or application records; decode or frame them separately. The binary I/O reference documents the returned count and blocking behavior.
Clean up
Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.
rm -- valid_buffer_prefix.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.