Skip to content
JavaScript: Detect duplicate keys after Unicode normalization

JavaScript: Detect duplicate keys after Unicode normalization

November 18, 2026

Two labels can display the same accented word while containing different Unicode sequences. If an application’s key policy treats canonically equivalent spellings as the same label, normalize before checking uniqueness. Retain the original spelling separately for display.

Set up

Tested on Debian 13 Linux with Node.js v24.19.0. Use Bash, Node.js on your PATH, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal through cleanup. Stop if setup fails before saving the source.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save as javascript-normalize-unique-keys.mjs:

import assert from 'node:assert/strict';

function normalizedKey(raw) {
  if (typeof raw !== 'string') throw new TypeError('Expected a string key');
  return raw.normalize('NFC');
}

function uniqueLabels(labels) {
  const originals = new Map();
  for (const label of labels) {
    const key = normalizedKey(label);
    if (originals.has(key)) throw new Error('Duplicate normalized key');
    originals.set(key, label);
  }
  return originals;
}

const composed = 'caf\u00e9';
const decomposed = 'cafe\u0301';
assert.notEqual(composed, decomposed);
assert.equal(normalizedKey(composed), normalizedKey(decomposed));
assert.throws(() => uniqueLabels([composed, decomposed]), /Duplicate/);
const labels = uniqueLabels([decomposed, 'tea']);
assert.equal(labels.get(normalizedKey(composed)), decomposed);
assert.equal(uniqueLabels([]).size, 0);
assert.throws(() => uniqueLabels(['tea', 'tea']), /Duplicate/);
assert.throws(() => normalizedKey(null), TypeError);
assert.notEqual(normalizedKey('A'), normalizedKey('a'));
assert.notEqual(normalizedKey('a'), normalizedKey('\u0430'));
assert.notEqual(normalizedKey('\uff21'), normalizedKey('A'));

console.log('Raw equality:', composed === decomposed);
console.log('NFC equality:', normalizedKey(composed) === normalizedKey(decomposed));
console.log('Canonical duplicate: rejected');
console.log('Original spelling retained:', labels.get('caf\u00e9') === decomposed);
console.log('Type, exact-duplicate, empty, and distinct-character checks: passed');

Run:

node javascript-normalize-unique-keys.mjs

Observed output:

Raw equality: false
NFC equality: true
Canonical duplicate: rejected
Original spelling retained: true
Type, exact-duplicate, empty, and distinct-character checks: passed

Check the key that will be stored

The two spellings use a precomposed accented letter and a letter followed by a combining accent. Their raw strings differ. After NFC normalization, they produce the same key, so the import rejects the duplicate instead of overwriting the first entry.

The Map stores normalized keys and original labels as values. Lookups must apply the same normalization policy as insertion. The example checks that looking up the composed form retrieves the original decomposed label. An empty input produces an empty Map; an exact duplicate is rejected too. The helper permits an empty string as a label, so add a separate rule if your application forbids it.

ECMAScript defines String.prototype.normalize with NFC, NFD, NFKC and NFKD forms. This example chooses NFC explicitly. Changing the form changes the matching policy and should be treated as a data migration if keys are already stored.

Keep normalization separate from identity rules

The assertions preserve case differences, Latin versus Cyrillic letters, and fullwidth versus ordinary Latin letters under NFC. Normalization does not make every visually similar string identical. It is not a complete policy for account names, authorization or spoofing prevention.

Apply any allowed-character, length and case rules deliberately. Trimming and case conversion change which labels collide, so choose those rules explicitly. If multiple processes can create labels concurrently, enforce uniqueness on the stored normalized key in the backing database as well; this local Map cannot coordinate those writes.

Clean up

Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.

rm -- javascript-normalize-unique-keys.mjs
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.

Last updated on