JavaScript Basics: Handle UTF-8 Chunks with TextEncoder and TextDecoder

October 20, 2026

Text and byte chunks do not have matching boundaries. A chunk can end halfway through a UTF-8 character, and a destination buffer may be too small for the next character. TextEncoder and TextDecoder provide explicit ways to handle both cases.

These examples were tested on Linux with Node.js v24.19.0. They use built-in APIs only, require no network access, and include assertions that check the results. These APIs are available globally in Node.js. The WHATWG Encoding Standard defines incremental decoding and error behavior; Node’s utility reference documents its implementation.

Prepare a scratch directory

You need Bash, a text editor, Node.js 24 on your PATH, and the mktemp, rm, and rmdir utilities. Keep using the same terminal. Save each named source file as UTF-8; .mjs selects ES modules. Each example runs independently.

node --version
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

The tested runtime prints:

v24.19.0

1. Encode into limited space

Save this as encode-into-byte-storage.mjs:

import assert from 'node:assert/strict';

const encoder = new TextEncoder();
const text = 'A🍵B';
const first = new Uint8Array(4);
const progress = encoder.encodeInto(text, first);
assert.deepEqual(progress, { read: 1, written: 1 });
assert.deepEqual(Array.from(first), [65, 0, 0, 0]);

const remaining = text.slice(progress.read);
const second = new Uint8Array(5);
const rest = encoder.encodeInto(remaining, second);
assert.deepEqual(rest, { read: 3, written: 5 });
assert.equal(new TextDecoder().decode(second), '🍵B');
assert.deepEqual(encoder.encodeInto('🍵', new Uint8Array(3)),
  { read: 0, written: 0 });
console.log(`First pass: read ${progress.read}, wrote ${progress.written}`);
console.log(`Second pass: read ${rest.read}, wrote ${rest.written}`);
console.log(new TextDecoder().decode(second));

Run it:

node encode-into-byte-storage.mjs

Output:

First pass: read 1, wrote 1
Second pass: read 3, wrote 5
🍵B

read counts consumed UTF-16 code units; written counts UTF-8 bytes. The first destination has only three bytes left after A, so the four-byte emoji is left for another call. Use read to advance the source and use only the written portion of a destination. A loop must handle zero progress, usually by providing more space. TextEncoder always writes UTF-8.

2. Decode characters split across chunks

Save this as decode-split-utf8.mjs:

import assert from 'node:assert/strict';

const chunks = [
  Uint8Array.from([0xe2, 0x82]),
  Uint8Array.from([0xac, 0x20, 0xf0, 0x9f]),
  Uint8Array.from([0x8d, 0xb5]),
];
const decoder = new TextDecoder('utf-8', { fatal: true });
const pieces = chunks.map(chunk => decoder.decode(chunk, { stream: true }));
pieces.push(decoder.decode());
const text = pieces.join('');
assert.deepEqual(pieces, ['', '€ ', '🍵', '']);
assert.equal(text, '€ 🍵');
assert.equal(new TextDecoder().decode(new Uint8Array()), '');
console.log(JSON.stringify(pieces));
console.log(text);

Run it:

node decode-split-utf8.mjs

Output:

["","€ ","🍵",""]
€ 🍵

Keep one decoder for the whole byte sequence. stream: true retains incomplete characters between calls. The final call without that option flushes buffered input and ends the sequence, even when it returns an empty string. This test splits both a euro sign and an emoji; decoding each chunk with a fresh decoder would lose that continuity.

3. Reject malformed and unfinished input

Save this as reject-invalid-utf8.mjs:

import assert from 'node:assert/strict';

const malformed = Uint8Array.from([0xc3, 0x28]);
const repaired = new TextDecoder('utf-8').decode(malformed);
assert.equal(repaired, '\ufffd(');
assert.throws(() => new TextDecoder('utf-8', { fatal: true })
  .decode(malformed), TypeError);

const decoder = new TextDecoder('utf-8', { fatal: true });
const pending = decoder.decode(Uint8Array.from([0xe2, 0x82]), { stream: true });
assert.equal(pending, '');
assert.throws(() => decoder.decode(), TypeError);
assert.equal(new TextDecoder('utf-8', { fatal: true })
  .decode(Uint8Array.from([0x4f, 0x4b])), 'OK');
console.log(`Replacement decoding: ${JSON.stringify(repaired)}`);
console.log('Malformed bytes rejected');
console.log('Unfinished final character rejected');

Run it:

node reject-invalid-utf8.mjs

Output:

Replacement decoding: "�("
Malformed bytes rejected
Unfinished final character rejected

The default decoder replaces malformed input with the replacement character, U+FFFD. Set fatal: true when damaged input should fail instead. An unfinished character becomes an error only when the sequence ends, which is why the final flush matters. Successful decoding checks the byte encoding; it does not validate a document schema, make text safe for HTML, or establish that a sender is trustworthy.

Clean up

In the same terminal, remove only the named example files, return to your original directory, and remove the empty scratch directory:

rm -- encode-into-byte-storage.mjs decode-split-utf8.mjs reject-invalid-utf8.mjs
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.

Last updated on October 20, 2026

JavaScript Basics: Join Human-Readable Lists with Intl.ListFormat JavaScript Basics: Count and Cut Text with Intl.Segmenter