Python utf-8-sig: remove a leading BOM without deleting interior text
A UTF-8 file may start with the three-byte byte order mark. Decode with utf-8-sig when that optional signature is allowed, instead of deleting every U+FEFF character after decoding.
Set up
Tested on Linux with CPython 3.12.14. Use Python 3.12 or later, Bash, a text editor, mktemp, rm and rmdir. The example needs no network access. Run the steps in the same terminal and stop if setup fails. Run Python without -O so the assertions stay enabled.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save this as decode_utf8_signature.py:
from codecs import BOM_UTF8
payload = BOM_UTF8 + "name\nA\ufeffB\n".encode("utf-8")
plain = payload.decode("utf-8")
decoded = payload.decode("utf-8-sig")
print("plain UTF-8:", repr(plain))
print("signature-aware:", repr(decoded))
print("no signature:", repr(b"name\n".decode("utf-8-sig")))
print("global removal:", repr(decoded.replace("\ufeff", "")))
try:
(BOM_UTF8 + b"\xff").decode("utf-8-sig")
except UnicodeDecodeError:
print("invalid UTF-8: rejected")
assert decoded == "name\nA\ufeffB\n"
assert plain.startswith("\ufeff")Run:
python3 -B decode_utf8_signature.pyObserved output:
plain UTF-8: '\ufeffname\nA\ufeffB\n'
signature-aware: 'name\nA\ufeffB\n'
no signature: 'name\n'
global removal: 'name\nAB\n'
invalid UTF-8: rejectedRemove the signature only at the beginning
The utf-8-sig codec documentation specifies that decoding skips an initial UTF-8 signature when present. Bytes without that signature still decode normally. The interior U+FEFF in the example remains part of the text.
A global string replacement would change the interior field from A\ufeffB to AB. Whether that interior character is acceptable is a separate content-validation decision. The decoding default remains strict, so the invalid byte is rejected even after a valid signature.
Limits
This example decodes one complete byte sequence. For chunked input, keep one incremental decoder for the stream instead of decoding each chunk independently, since characters and the initial signature can cross chunk boundaries. Encoding with utf-8-sig adds a signature; choose plain utf-8 when writing a format that must omit it. The codec does not detect arbitrary encodings or normalize text.
Clean up
Remove the example file and the temporary directory:
rm -- decode_utf8_signature.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.