Python Basics: Inspecting Text with unicodedata

Two strings can look alike while containing different Unicode code points. Python’s unicodedata module helps inspect characters and normalize equivalent spellings.

Use Python 3.12 or newer on Linux. Each example runs independently with the standard library and only prints sample data.

Inspect a character

Save as demo_unicodedata_1.py:

import unicodedata

character = "é"
print(f"U+{ord(character):04X}")
print(unicodedata.name(character))
print(unicodedata.category(character))
print(unicodedata.name("\n", "No assigned name"))

Run it:

python3 demo_unicodedata_1.py

Output:

U+00E9
LATIN SMALL LETTER E WITH ACUTE
Ll
No assigned name

name() describes a character, and category() returns its general category. Here, Ll means a lowercase letter. Providing a default to name() handles characters without an assigned name, such as the newline.

Normalize before comparing

Save as demo_unicodedata_2.py:

import unicodedata

composed_text = "caf\u00e9"
decomposed_text = "cafe\u0301"
print(composed_text == decomposed_text)
print(len(composed_text), len(decomposed_text))
normalized_text = unicodedata.normalize("NFC", decomposed_text)
print(composed_text == normalized_text)
print(unicodedata.is_normalized("NFC", normalized_text))

Run it:

python3 demo_unicodedata_2.py

Output:

False
4 5
True
True

The second spelling uses e followed by a combining accent. NFC puts canonically equivalent text into a composed form where possible. Apply the same normalization policy to both inputs; this example’s first input is already NFC. The original strings remain unchanged.

Read numeric character properties

Save as demo_unicodedata_3.py:

import unicodedata

arabic_indic_digit = "٣"
fraction_character = "½"
print(unicodedata.decimal(arabic_indic_digit))
print(unicodedata.numeric(fraction_character))
print(unicodedata.decimal(fraction_character, None))

Run it:

python3 demo_unicodedata_3.py

Output:

3
0.5
None

decimal() reads a decimal digit’s integer value. numeric() also recognizes characters such as fractions and returns a float. These functions inspect one character, rather than parsing a whole number string.

Normalization is not universal identity validation. It does not make every visually similar character equal or detect impersonation. NFKC also folds compatibility distinctions, so choose a form to match your application’s text rules.

References

• Python unicodedata documentation

Acknowledgements

This post was created with assistance from AI.