JavaScript: Detect duplicate keys after Unicode normalization
Two labels can display the same accented word while containing different Unicode sequences. If an application’s key policy treats canonically equivalent spellings as the same label, normalize before checking uniqueness. Retain the original spelling separately for display.
Set up
Tested on Debian 13 Linux with Node.js v24.19.0. Use Bash, Node.js on your PATH, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal through cleanup. Stop if setup fails before saving the source.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as javascript-normalize-unique-keys.mjs:
import assert from 'node:assert/strict';
function normalizedKey(raw) {
if (typeof raw !== 'string') throw new TypeError('Expected a string key');
return raw.normalize('NFC');
}
function uniqueLabels(labels) {
const originals = new Map();
for (const label of labels) {
const key = normalizedKey(label);
if (originals.has(key)) throw new Error('Duplicate normalized key');
originals.set(key, label);
}
return originals;
}
const composed = 'caf\u00e9';
const decomposed = 'cafe\u0301';
assert.notEqual(composed, decomposed);
assert.equal(normalizedKey(composed), normalizedKey(decomposed));
assert.throws(() => uniqueLabels([composed, decomposed]), /Duplicate/);
const labels = uniqueLabels([decomposed, 'tea']);
assert.equal(labels.get(normalizedKey(composed)), decomposed);
assert.equal(uniqueLabels([]).size, 0);
assert.throws(() => uniqueLabels(['tea', 'tea']), /Duplicate/);
assert.throws(() => normalizedKey(null), TypeError);
assert.notEqual(normalizedKey('A'), normalizedKey('a'));
assert.notEqual(normalizedKey('a'), normalizedKey('\u0430'));
assert.notEqual(normalizedKey('\uff21'), normalizedKey('A'));
console.log('Raw equality:', composed === decomposed);
console.log('NFC equality:', normalizedKey(composed) === normalizedKey(decomposed));
console.log('Canonical duplicate: rejected');
console.log('Original spelling retained:', labels.get('caf\u00e9') === decomposed);
console.log('Type, exact-duplicate, empty, and distinct-character checks: passed');Run:
node javascript-normalize-unique-keys.mjsObserved output:
Raw equality: false
NFC equality: true
Canonical duplicate: rejected
Original spelling retained: true
Type, exact-duplicate, empty, and distinct-character checks: passedCheck the key that will be stored
The two spellings use a precomposed accented letter and a letter followed by a combining accent. Their raw strings differ. After NFC normalization, they produce the same key, so the import rejects the duplicate instead of overwriting the first entry.
The Map stores normalized keys and original labels as values. Lookups must apply the same normalization policy as insertion. The example checks that looking up the composed form retrieves the original decomposed label. An empty input produces an empty Map; an exact duplicate is rejected too. The helper permits an empty string as a label, so add a separate rule if your application forbids it.
ECMAScript defines String.prototype.normalize with NFC, NFD, NFKC and NFKD forms. This example chooses NFC explicitly. Changing the form changes the matching policy and should be treated as a data migration if keys are already stored.
Keep normalization separate from identity rules
The assertions preserve case differences, Latin versus Cyrillic letters, and fullwidth versus ordinary Latin letters under NFC. Normalization does not make every visually similar string identical. It is not a complete policy for account names, authorization or spoofing prevention.
Apply any allowed-character, length and case rules deliberately. Trimming and case conversion change which labels collide, so choose those rules explicitly. If multiple processes can create labels concurrently, enforce uniqueness on the stored normalized key in the backing database as well; this local Map cannot coordinate those writes.
Clean up
Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.
rm -- javascript-normalize-unique-keys.mjs
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.