Reject duplicate CSV headers before building dictionaries
A CSV export with two columns named quantity loses one value when read into a dictionary. Validate the header before reading records so a repeated name cannot silently replace an earlier field.
Set up
Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as checked_headers.py:
import csv
from io import StringIO
def read_rows(text):
reader = csv.DictReader(StringIO(text, newline=""), strict=True)
names = reader.fieldnames
if names is None:
raise ValueError("missing header")
if len(names) != len(set(names)):
raise ValueError("duplicate header")
if set(names) != {"name", "quantity"}:
raise ValueError("expected name and quantity")
rows = list(reader)
if any(None in row or None in row.values() for row in rows):
raise ValueError("row width differs from header")
return rows
bad = "name,quantity,quantity\nPencil,2,99\n"
unchecked = next(csv.DictReader(StringIO(bad)))
print("Unchecked:", unchecked)
try:
read_rows(bad)
except ValueError as error:
print("Rejected:", error)
else:
raise AssertionError("duplicate header was accepted")
good = read_rows("quantity,name\n2,Pencil\n")
assert good == [{"quantity": "2", "name": "Pencil"}]
print("Checked:", good)Run:
python3 -B checked_headers.pyObserved output:
Unchecked: {'name': 'Pencil', 'quantity': '99'}
Rejected: duplicate header
Checked: [{'quantity': '2', 'name': 'Pencil'}]Validate before conversion
The uniqueness check runs before any data row becomes a dictionary. The schema accepts either column order, so comparing only the set of names would be insufficient: sets also hide repeated names. Checking the length first preserves that distinction.
DictReader puts surplus fields under a None key and fills missing fields with None. The row-width check rejects both. Empty cells remain empty strings and are accepted here.
Keep application rules separate
Header spelling is exact and case-sensitive. Decide explicitly whether your importer should trim spaces or normalize case; if you normalize names, check for duplicates after normalization too. This example retains strings and does not validate quantities or blank names.
The fixture is small and in memory. A larger importer needs size limits and a policy for whether to buffer all validated records before applying changes. For real files, use an explicit encoding and newline="". The CSV reference documents header selection and unmatched row fields.
Clean up
Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.
rm -- checked_headers.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.