Python Basics: Working with In-Memory Streams with io

Sometimes a function expects a file-like object, but your data is already in memory. Python’s io.StringIO holds text, while io.BytesIO holds bytes. Both provide familiar stream methods without needing a data file.

Before you start

These examples were tested on Linux with Python 3.12.14 and 3.13.5, using Bash and only the standard library. Make sure python3 runs one of those Python versions. The three scripts are independent. They keep their sample data in memory, print results, and do not use the network.

Create a fresh directory for the example scripts. Keep this Bash session open for cleanup:

start_dir="$PWD"
demo_dir="$(mktemp -d)" || exit 1
cd "$demo_dir" || exit 1

Save the following UTF-8 source files in this directory, then run each command there. The setup creates the directory; saving the scripts creates the only files.

Write text, then move the cursor

Save as demo_io_text.py:

from io import StringIO

with StringIO() as stream:
    stream.write("alpha\nbeta\n")
    print("Position:", stream.tell())
    print("At end:", repr(stream.read()))

    stream.seek(0)
    print("First line:", repr(stream.readline()))
    print("Remaining:", repr(stream.read()))
    print("Whole value:", repr(stream.getvalue()))

Run it:

python3 demo_io_text.py

Output:

Position: 11
At end: ''
First line: 'alpha\n'
Remaining: 'beta\n'
Whole value: 'alpha\nbeta\n'

Writing advances the cursor. Reading at the end returns an empty string; seek(0) returns to the beginning. getvalue() retrieves the whole buffer without moving the cursor. repr() makes newlines visible. Retrieve anything you need before the with block ends, because closing the stream discards its buffer.

Keep bytes separate from text

Save as demo_io_bytes.py:

from io import BytesIO

with BytesIO() as stream:
    stream.write("café".encode("utf-8"))
    stream.seek(0)
    data = stream.read()

    print("Bytes:", data)
    print("Byte count:", len(data))
    print("Decoded:", data.decode("utf-8"))

    try:
        stream.write("more text")
    except TypeError as error:
        print("Expected error:", type(error).__name__)

Run it:

python3 demo_io_bytes.py

Output:

Bytes: b'caf\xc3\xa9'
Byte count: 5
Decoded: café
Expected error: TypeError

The four-character word occupies five bytes in UTF-8. BytesIO does not encode or decode text for you. The final write deliberately passes a string and catches the resulting TypeError, so the script exits normally. Conversely, StringIO expects strings, not bytes.

Round-trip CSV without a data file

Save as demo_io_csv.py:

import csv
from io import StringIO

rows = [
    ["item", "note"],
    ["tea", "hot, please"],
    ["bread", "first line\nsecond line"],
]

with StringIO(newline="") as stream:
    writer = csv.writer(stream, lineterminator="\n")
    writer.writerows(rows)
    print("CSV:", repr(stream.getvalue()))

    stream.seek(0)
    restored = list(csv.reader(stream))
    print("Round-trip:", restored == rows)
    print("Last note:", repr(restored[2][1]))

Run it:

python3 demo_io_csv.py

Output:

CSV: 'item,note\ntea,"hot, please"\nbread,"first line\nsecond line"\n'
Round-trip: True
Last note: 'first line\nsecond line'

The CSV writer quotes fields containing commas or newlines. newline="" leaves newline handling to the CSV module, and the explicit line terminator makes this example’s output predictable. Rewind before reading. A CSV reader understands the embedded newline, which a simple line-by-line split would mishandle.

In-memory streams are useful for small conversions and tests. Their contents still consume memory, so use a real file or a streaming approach for data too large to keep in RAM.

Clean up

After running all three examples, remove only their source files and the now-empty temporary directory:

rm -- demo_io_text.py demo_io_bytes.py demo_io_csv.py
cd "$start_dir" || exit 1
rmdir -- "$demo_dir"
unset start_dir demo_dir

References

• Python io documentation
• Python csv documentation

Acknowledgements

This post was created with assistance from AI.