Linux Basics: Convert Text Encodings Without Accepting Partial Output

A legacy text import may use ISO-8859-1 while the next tool expects UTF-8. iconv converts between declared encodings; it does not discover the correct source encoding. A failed conversion can leave partial output, so write to a separate candidate and check success before using it.

Prepare a scratch directory

Tested on Linux with GNU Bash 5.2.37 and GNU coreutils 9.7, with glibc iconv 2.41. You need Bash, a text editor, and the utilities shown below. The script creates and removes its own temporary fixture directory.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as demonstrate-iconv-strict-conversion.sh:

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
fixture_dir="$(mktemp -d)"
trap 'rm -rf -- "$fixture_dir"' EXIT
cd -- "$fixture_dir"
printf 'caf\351\n' > latin1.txt
if iconv -f ISO-8859-1 -t UTF-8 latin1.txt > converted.part; then
    mv -- converted.part converted.txt
else
    rm -f -- converted.part
    exit 1
fi
printf 'UTF-8 bytes:\n'
od -An -tx1 converted.txt
iconv -f UTF-8 -t ISO-8859-1 converted.txt > roundtrip.txt
cmp -- latin1.txt roundtrip.txt
printf 'Round trip matches: yes\n'
printf 'good\377bad\n' > invalid.txt
if iconv -f UTF-8 -t UTF-8 invalid.txt > rejected.part 2> conversion-error.txt; then
    printf 'Unexpected success\n' >&2
    exit 1
else
    printf 'Invalid UTF-8 rejected; candidate discarded\n'
    rm -- rejected.part
fi
test -s conversion-error.txt

Run it:

bash demonstrate-iconv-strict-conversion.sh

Observed output:

UTF-8 bytes:
 63 61 66 c3 a9 0a
Round trip matches: yes
Invalid UTF-8 rejected; candidate discarded

Read the result

The input byte e9 represents é in ISO-8859-1. Its UTF-8 representation is c3 a9, which the byte dump makes explicit without depending on terminal rendering. Converting back reproduces the original bytes. The final fixture deliberately includes ff, which is not valid UTF-8. Its nonzero status takes the failure branch, and the candidate is removed instead of being accepted as a complete conversion.

Apply it carefully

Never redirect conversion output onto the input file: the shell can truncate the input before the converter reads it. The example keeps both the source and successful output. Obtain the source encoding from the producing system or a documented file format; a successful conversion alone does not prove it was the correct encoding. Avoid -c, //IGNORE, or //TRANSLIT when exact preservation is required, since they permit dropping or approximating characters. The same-directory rename shown here controls when this example uses its result; it does not provide crash durability or preserve original metadata.

The command reference documents the available options.

Clean up

Remove the saved script, return to the starting directory, and remove the empty practice directory:

rm -- demonstrate-iconv-strict-conversion.sh
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect what remains rather than deleting unknown files.

Acknowledgements

This post was created with assistance from AI.