Go: Validate UTF-8 before counting runes
A Go string can contain arbitrary bytes. Counting its runes does not establish that those bytes are valid UTF-8: malformed encodings are counted as replacement runes. Validate the encoding first when invalid input must be rejected.
Set up
Tested on Linux amd64 with Go 1.26.10. Use Bash with Go on your PATH, a text editor, and the mktemp, rm, and rmdir utilities. The example uses only the standard library. Keep the same terminal through cleanup, and stop if setup fails before saving the source.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as go-utf8-validate-before-rune-count.go:
package main
import (
"fmt"
"unicode/utf8"
)
func countText(text string) (int, error) {
if !utf8.ValidString(text) {
return 0, fmt.Errorf("invalid UTF-8")
}
return utf8.RuneCountInString(text), nil
}
func main() {
samples := []struct {
label, text string
valid bool
}{
{"ASCII", "cat", true},
{"accent", "é", true},
{"combining", "e\u0301", true},
{"replacement", "\ufffd", true},
{"invalid byte", string([]byte{0xff}), false},
{"truncated", string([]byte{0xe2, 0x82}), false},
{"empty", "", true},
}
for _, sample := range samples {
count, err := countText(sample.text)
if (err == nil) != sample.valid {
panic("encoding classification")
}
fmt.Printf("%s: bytes=%d runes=%d error=%v\n", sample.label, len(sample.text), count, err)
}
}Run:
go run go-utf8-validate-before-rune-count.goObserved output:
ASCII: bytes=3 runes=3 error=<nil>
accent: bytes=2 runes=1 error=<nil>
combining: bytes=3 runes=2 error=<nil>
replacement: bytes=3 runes=1 error=<nil>
invalid byte: bytes=1 runes=0 error=invalid UTF-8
truncated: bytes=2 runes=0 error=invalid UTF-8
empty: bytes=0 runes=0 error=<nil>Read the result
A correctly encoded replacement character is valid text, so rejecting every occurrence of that character would be incorrect. The byte validator checks the encoding itself. The truncated multibyte sequence fails before the helper returns a usable count.
The composed accent and the combining sequence can look similar while containing different numbers of runes. The output also shows that byte length and rune count measure different things.
Apply it carefully
A rune count is not a count of displayed characters or grapheme clusters. Valid UTF-8 can still contain control characters, combining marks or visually confusing text. Apply any application-level character restrictions and normalization separately. This helper scans an already-loaded string and does not bound input bytes. For streamed data, a chunk may end inside a valid multibyte sequence; preserve incomplete bytes across chunks instead of rejecting each chunk independently.
See the Go standard-library reference.
Clean up
Remove the saved source and the empty scratch directory. Go manages its build cache separately. If the directory is not empty, inspect its contents before removing anything else.
rm -- go-utf8-validate-before-rune-count.go
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.