Preserve the boundary item when splitting an iterator prefix
A parser may read comment headers before handing the remaining rows to a CSV reader. itertools.takewhile() consumes the first item that fails its predicate. If that item is the CSV header, continuing from the original iterator silently skips it.
Set up
Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as preserve_iterator_boundary.py:
from itertools import chain, takewhile
def split_prefix(predicate, iterable):
source = iter(iterable)
prefix = []
for item in source:
if not predicate(item):
return prefix, chain((item,), source)
prefix.append(item)
return prefix, iter(())
lines = ["# owner=ops", "# format=v1", "name,quantity", "pen,2"]
source = iter(lines)
headers = list(takewhile(lambda line: line.startswith("#"), source))
print("takewhile headers:", headers)
print("takewhile remainder:", list(source))
headers, body = split_prefix(lambda line: line.startswith("#"), lines)
body = list(body)
assert headers == lines[:2]
assert body == lines[2:]
print("Preserved headers:", headers)
print("Preserved remainder:", body)
for values in ([], ["name,quantity"], ["# only a header"]):
prefix, remainder = split_prefix(lambda line: line.startswith("#"), values)
assert prefix + list(remainder) == values
print("Empty, no-prefix, all-prefix: passed")Run:
python3 -B preserve_iterator_boundary.pyObserved output:
takewhile headers: ['# owner=ops', '# format=v1']
takewhile remainder: ['pen,2']
Preserved headers: ['# owner=ops', '# format=v1']
Preserved remainder: ['name,quantity', 'pen,2']
Empty, no-prefix, all-prefix: passedReturn the first rejected item with the remainder
The first pass loses the column-header row: it has already been read to decide that the comment prefix ended. The takewhile reference documents that consumption.
The helper uses the same test but keeps the rejected item. A chained iterator yields it first, then continues with the original source. The next parser therefore receives the column header before the data row. The boundary tests also preserve the full input when it is empty, starts with data, or contains only comments.
Once the helper returns, consume its returned remainder. Reading separately from the original iterator would advance the shared source behind that remainder. The returned iterator is also one-shot; reading it twice does not replay the rows.
Bound the prefix when input is large
The helper stores matching items in a list and leaves the remainder lazy. Memory grows with the length of the matching prefix. An unbounded source that never fails the predicate would never return. Add a record-count or byte limit when the input contract does not already bound the prefix.
The predicate can have effects or raise, and the item being tested has already been obtained from the source. This helper does not rewind the source on exceptions. It assumes string rows and uses a fixed comment marker only to demonstrate the handoff; a complete format parser must define which rows count as comments.
Clean up
Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.
rm -- preserve_iterator_boundary.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.