Skip to content
Python utf-8-sig: remove a leading BOM without deleting interior text

Python utf-8-sig: remove a leading BOM without deleting interior text

November 20, 2026

A UTF-8 file may start with the three-byte byte order mark. Decode with utf-8-sig when that optional signature is allowed, instead of deleting every U+FEFF character after decoding.

Set up

Tested on Linux with CPython 3.12.14. Use Python 3.12 or later, Bash, a text editor, mktemp, rm and rmdir. The example needs no network access. Run the steps in the same terminal and stop if setup fails. Run Python without -O so the assertions stay enabled.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as decode_utf8_signature.py:

from codecs import BOM_UTF8

payload = BOM_UTF8 + "name\nA\ufeffB\n".encode("utf-8")
plain = payload.decode("utf-8")
decoded = payload.decode("utf-8-sig")
print("plain UTF-8:", repr(plain))
print("signature-aware:", repr(decoded))
print("no signature:", repr(b"name\n".decode("utf-8-sig")))
print("global removal:", repr(decoded.replace("\ufeff", "")))
try:
    (BOM_UTF8 + b"\xff").decode("utf-8-sig")
except UnicodeDecodeError:
    print("invalid UTF-8: rejected")
assert decoded == "name\nA\ufeffB\n"
assert plain.startswith("\ufeff")

Run:

python3 -B decode_utf8_signature.py

Observed output:

plain UTF-8: '\ufeffname\nA\ufeffB\n'
signature-aware: 'name\nA\ufeffB\n'
no signature: 'name\n'
global removal: 'name\nAB\n'
invalid UTF-8: rejected

Remove the signature only at the beginning

The utf-8-sig codec documentation specifies that decoding skips an initial UTF-8 signature when present. Bytes without that signature still decode normally. The interior U+FEFF in the example remains part of the text.

A global string replacement would change the interior field from A\ufeffB to AB. Whether that interior character is acceptable is a separate content-validation decision. The decoding default remains strict, so the invalid byte is rejected even after a valid signature.

Limits

This example decodes one complete byte sequence. For chunked input, keep one incremental decoder for the stream instead of decoding each chunk independently, since characters and the initial signature can cross chunk boundaries. Encoding with utf-8-sig adds a signature; choose plain utf-8 when writing a format that must omit it. The codec does not detect arbitrary encodings or normalize text.

Clean up

Remove the example file and the temporary directory:

rm -- decode_utf8_signature.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.

Last updated on