Skip to content
Process only the bytes returned by readinto

Process only the bytes returned by readinto

November 17, 2026

readinto() fills a buffer you provide and returns the number of bytes read. A short read leaves the rest of that buffer unchanged. Processing its entire capacity can append old bytes to the new data.

Set up

Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save as valid_buffer_prefix.py:

from io import BytesIO


payload = b"ABCDEFGHijk"
buffer = bytearray(8)
chunks = []

with BytesIO(payload) as source, memoryview(buffer) as view:
    while True:
        count = source.readinto(buffer)
        if count == 0:
            break
        valid = bytes(view[:count])
        chunks.append(valid)
        print("Read:", count, "buffer:", bytes(buffer), "valid:", valid)

restored = b"".join(chunks)
assert chunks == [b"ABCDEFGH", b"ijk"]
assert restored == payload
assert bytes(buffer) == b"ijkDEFGH"
print("Restored:", restored)

Run:

python3 -B valid_buffer_prefix.py

Observed output:

Read: 8 buffer: b'ABCDEFGH' valid: b'ABCDEFGH'
Read: 3 buffer: b'ijkDEFGH' valid: b'ijk'
Restored: b'ABCDEFGHijk'

Separate capacity from valid length

The buffer keeps its eight-byte capacity throughout the loop. On the second read, only its first three bytes are replaced. The returned count identifies the valid prefix; clearing the tail is unnecessary when every consumer respects that count.

bytes(view[:count]) copies the prefix before the buffer is reused. Saving the view itself would keep a reference to mutable storage, so earlier chunks could change on the next read. Immediate consumers can use a view without retaining it, provided they finish before the buffer changes.

Keep stream handling explicit

The example uses a blocking in-memory binary stream and a nonempty buffer. Zero signals EOF here. A short positive read must still be processed; do not discard it or assume every call fills the buffer. Nonblocking streams need separate would-block handling, which can involve None or BlockingIOError depending on the interface.

Collecting all chunks retains the full payload in memory. A streaming application can process each valid prefix immediately instead. Byte chunks also need not align with text characters or application records; decode or frame them separately. The binary I/O reference documents the returned count and blocking behavior.

Clean up

Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.

rm -- valid_buffer_prefix.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.

Last updated on