JavaScript Basics: Decode UTF-8 Across Buffer Boundaries

A byte stream can split a character between chunks. Decoding each buffer separately then joining the strings loses that character. StringDecoder keeps the incomplete bytes until the next chunk arrives, which is useful when reading text from Node streams.

Prepare a scratch directory

Tested on Linux with Node.js v24.19.0. You need Node, Bash, a text editor, and the standard mktemp, rm, and rmdir utilities. No npm packages or remote services are needed. The .mjs extension selects ES modules.

Keep using the same terminal. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as decode-chunks.mjs:

import assert from 'node:assert/strict';
import { StringDecoder } from 'node:string_decoder';

const bytes = Buffer.from('A€B', 'utf8');
const chunks = [bytes.subarray(0, 2), bytes.subarray(2, 3), bytes.subarray(3)];
const decoder = new StringDecoder('utf8');
const parts = chunks.map(chunk => decoder.write(chunk));
parts.push(decoder.end());
assert.deepEqual(parts, ['A', '', '€B', '']);
assert.equal(parts.join(''), 'A€B');
console.log('Decoded pieces:', JSON.stringify(parts));
console.log('Joined:', parts.join(''));

const broken = chunks.map(chunk => chunk.toString('utf8')).join('');
assert.notEqual(broken, 'A€B');
console.log('Separate decoding loses data:', broken !== 'A€B');

const truncated = new StringDecoder('utf8');
assert.equal(truncated.write(Buffer.from([0xe2, 0x82])), '');
assert.equal(truncated.end(), '\uFFFD');
console.log('Incomplete final character: replacement emitted');

Run it:

node decode-chunks.mjs

Observed output:

Decoded pieces: ["A","","€B",""]
Joined: A€B
Separate decoding loses data: true
Incomplete final character: replacement emitted

Read the result

The first chunk ends after the first byte of the euro sign; the second contains only its next byte. Neither can complete the character. The third chunk supplies its last byte and the final B. The empty strings are expected, so do not treat an empty decoded piece as end-of-stream.

Call end() when the input finishes to flush the decoder. The truncated fixture shows that this decoder replaces incomplete UTF-8 with the replacement character. It does not provide strict validation. Use a fatal TextDecoder when malformed data must be rejected. A decoder also does not split lines or preserve message boundaries; that is a separate parsing step.

The official Node.js reference describes the API. The assertions above check the specific fixture and its intended edge case.

Clean up

Remove the source and then the empty practice directory:

rm -- decode-chunks.mjs
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect the remaining files rather than deleting unknown contents.

Acknowledgements

This post was created with assistance from AI.