Python Basics: Encoding Text and Decoding Byte Chunks

Text and bytes represent different things. Encode text when a boundary needs bytes, and decode bytes using the format that produced them. Invalid input and split multibyte characters need deliberate handling.

These three standalone examples were tested on Linux with Python 3.12.14. They require no third-party packages. Assertions check the results; run without Python’s -O option.

Prepare a scratch directory

You need Bash, Python 3.12 or newer on your PATH, a text editor, and the mktemp, rm, and rmdir utilities. Keep using the same terminal and save each named source file as UTF-8.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

1. Inspect an explicit UTF-8 conversion

Save as inspect_utf8_encoding.py:

message = "A€"
payload = message.encode("utf-8")
restored = payload.decode("utf-8")

assert payload == b"A\xe2\x82\xac"
assert restored == message
assert payload[0] == 65
assert payload[:1] == b"A"
print("Code points:", len(message))
print("Bytes:", len(payload))
print("Hex:", payload.hex())
print("First byte:", payload[0])
print("Decoded:", restored)

Run it:

python3 inspect_utf8_encoding.py

Output:

Code points: 2
Bytes: 4
Hex: 41e282ac
First byte: 65
Decoded: A€

The euro sign occupies one Unicode code point and three UTF-8 bytes. A bytes index gives an integer; a bytes slice gives bytes. Use decode to recover text. str(payload) without an encoding produces a representation of the bytes, not the decoded message.

2. Choose what to do with invalid input

Save as handle_invalid_utf8.py:

payload = b"batch:\xff"
try:
    payload.decode("utf-8")
except UnicodeDecodeError as error:
    assert error.start == 6
    print("Invalid byte at index:", error.start)
else:
    raise AssertionError("Expected invalid UTF-8")

replaced = payload.decode("utf-8", errors="replace")
ignored = payload.decode("utf-8", errors="ignore")
assert replaced == "batch:\ufffd"
assert ignored == "batch:"
print("Replacement text:", ascii(replaced))
print("Ignored byte:", ascii(ignored))

Run it:

python3 handle_invalid_utf8.py

Output:

Invalid byte at index: 6
Replacement text: 'batch:\ufffd'
Ignored byte: 'batch:'

Strict decoding is the default and exposes malformed input. replace inserts a replacement character; ignore drops the bad byte. Both alternatives lose information. Use them only when that loss is acceptable, such as an explicitly best-effort display. ascii makes the replacement character visible as an escape.

3. Decode a character split across chunks

Save as decode_utf8_chunks.py:

import codecs

decoder = codecs.getincrementaldecoder("utf-8")()
chunks = [b"price:\xe2", b"\x82", b"\xac"]
pieces = []
for chunk in chunks:
    piece = decoder.decode(chunk)
    pieces.append(piece)
    print("Part:", ascii(piece))
pieces.append(decoder.decode(b"", final=True))
result = "".join(pieces)
assert result == "price:€"
print("Complete:", ascii(result))

incomplete = codecs.getincrementaldecoder("utf-8")()
assert incomplete.decode(b"\xe2") == ""
try:
    incomplete.decode(b"", final=True)
except UnicodeDecodeError:
    print("Incomplete final character rejected")
else:
    raise AssertionError("Expected incomplete UTF-8")

Run it:

python3 decode_utf8_chunks.py

Output:

Part: 'price:'
Part: ''
Part: '\u20ac'
Complete: 'price:\u20ac'
Incomplete final character rejected

An incremental decoder retains an unfinished character between calls. Feed chunks from the same byte stream into the same decoder. final=True signals the end and catches a trailing incomplete character. Independently decoding each chunk would reject the split euro sign.

When to use this

Agree on an encoding at each input/output boundary instead of guessing from appearance. Keep raw bytes when exact recovery matters. A successful decode alone does not prove that the chosen encoding matches the producer.

References

• Unicode HOWTO: text and encodings
• Incremental encoding and decoding

Clean up

In the same terminal, remove only these example files, then return to your original directory and remove the empty scratch directory:

rm -- inspect_utf8_encoding.py handle_invalid_utf8.py decode_utf8_chunks.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.