Go Basics: Set Explicit Limits for Line-Oriented Input

Log processors often work until one unusually large line arrives. Configure a small explicit limit in a fixture and verify that stopping on an oversized token is reported as an error.

Prepare a scratch directory

Tested on Linux amd64 with Go 1.26.10. Install Go and make go available on PATH. You also need Bash, a text editor, and the standard mktemp, rm, and rmdir utilities. The example uses only the standard library and needs no module or external dependency.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as scanner-limits-demo.go:

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func scan(input string) (int, error) {
	scanner := bufio.NewScanner(strings.NewReader(input))
	scanner.Buffer(make([]byte, 16), 64)
	count := 0
	for scanner.Scan() {
		count++
	}
	return count, scanner.Err()
}
func main() {
	n, err := scan("first\nsecond\n")
	fmt.Printf("Short lines: count=%d error=%t\n", n, err != nil)
	n, err = scan(strings.Repeat("x", 100) + "\n")
	fmt.Printf("Oversized line: count=%d error=%t\n", n, err != nil)
}

Run it:

go run scanner-limits-demo.go

Observed output:

Short lines: count=2 error=false
Oversized line: count=0 error=true

Read the result

The scanner accepts the short lines and stops before yielding the oversized line. Always inspect Err() after the loop: a false return from Scan() can mean either end of input or failure. Configuring Buffer before scanning gives the parser an explicit allocation ceiling for this fixture.

Apply it carefully

The maximum buffer size is not an exact accepted payload length because delimiters and buffering overhead can need space. Choose a limit with that overhead in mind. If you need to stream arbitrarily long records, use bufio.Reader with your own chunking and size policy instead of merely raising the scanner limit indefinitely.

The official Go reference documents the behavior used here.

Clean up

Remove the saved source, return to the starting directory, and remove the empty practice directory. go run does not leave an executable here; Go manages its build cache separately.

rm -- scanner-limits-demo.go
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect its remaining contents before removing anything else.

Acknowledgements

This post was created with assistance from AI.