Skip to content
Reject duplicate CSV headers before building dictionaries

Reject duplicate CSV headers before building dictionaries

November 17, 2026

A CSV export with two columns named quantity loses one value when read into a dictionary. Validate the header before reading records so a repeated name cannot silently replace an earlier field.

Set up

Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save as checked_headers.py:

import csv
from io import StringIO


def read_rows(text):
    reader = csv.DictReader(StringIO(text, newline=""), strict=True)
    names = reader.fieldnames
    if names is None:
        raise ValueError("missing header")
    if len(names) != len(set(names)):
        raise ValueError("duplicate header")
    if set(names) != {"name", "quantity"}:
        raise ValueError("expected name and quantity")
    rows = list(reader)
    if any(None in row or None in row.values() for row in rows):
        raise ValueError("row width differs from header")
    return rows


bad = "name,quantity,quantity\nPencil,2,99\n"
unchecked = next(csv.DictReader(StringIO(bad)))
print("Unchecked:", unchecked)

try:
    read_rows(bad)
except ValueError as error:
    print("Rejected:", error)
else:
    raise AssertionError("duplicate header was accepted")

good = read_rows("quantity,name\n2,Pencil\n")
assert good == [{"quantity": "2", "name": "Pencil"}]
print("Checked:", good)

Run:

python3 -B checked_headers.py

Observed output:

Unchecked: {'name': 'Pencil', 'quantity': '99'}
Rejected: duplicate header
Checked: [{'quantity': '2', 'name': 'Pencil'}]

Validate before conversion

The uniqueness check runs before any data row becomes a dictionary. The schema accepts either column order, so comparing only the set of names would be insufficient: sets also hide repeated names. Checking the length first preserves that distinction.

DictReader puts surplus fields under a None key and fills missing fields with None. The row-width check rejects both. Empty cells remain empty strings and are accepted here.

Keep application rules separate

Header spelling is exact and case-sensitive. Decide explicitly whether your importer should trim spaces or normalize case; if you normalize names, check for duplicates after normalization too. This example retains strings and does not validate quantities or blank names.

The fixture is small and in memory. A larger importer needs size limits and a policy for whether to buffer all validated records before applying changes. For real files, use an explicit encoding and newline="". The CSV reference documents header selection and unmatched row fields.

Clean up

Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.

rm -- checked_headers.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.

Last updated on