Keep Python groupby results before advancing the iterator
A log report that saves group iterators for later can lose the rows it meant to summarize. Consume each group while it is current, or save its contents before advancing to the next key.
Set up
Tested on Linux with CPython 3.12.14. Use Bash, Python 3.12 or newer, a text editor, and the mktemp, rm and rmdir utilities. No packages or network access are needed. Keep the same terminal open and stop if setup fails. Run without -O so the assertions remain enabled.
start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1Run the example
Save as group_snapshots.py:
from itertools import groupby
from operator import itemgetter
events = [("api", 12), ("api", 18), ("worker", 7), ("api", 9)]
by_service = itemgetter(0)
deferred = list(groupby(events, key=by_service))
print("Deferred:", [(name, list(rows)) for name, rows in deferred])
snapshots = [(name, list(rows)) for name, rows in groupby(events, key=by_service)]
print("Run totals:", [(name, sum(value for _, value in rows)) for name, rows in snapshots])
ordered = sorted(events, key=by_service)
totals = [
(name, sum(value for _, value in rows))
for name, rows in groupby(ordered, key=by_service)
]
print("Service totals:", totals)
assert [len(rows) for _, rows in snapshots] == [2, 1, 1]
assert totals == [("api", 39), ("worker", 7)]Run:
python3 -B group_snapshots.pyObserved output:
Deferred: [('api', []), ('worker', []), ('api', [])]
Run totals: [('api', 30), ('worker', 7), ('api', 9)]
Service totals: [('api', 39), ('worker', 7)]Choose runs or combined totals
The deferred list saves three group iterators but consumes none of their rows before advancing the outer iterator. By the time the later loop reads them, each is empty. The snapshot comprehension consumes each group immediately, preserving the two API runs separately.
The last calculation sorts by service before grouping, so all three API values contribute to one total. Sorting changes event order. Keep the original order when the runs themselves matter, such as transitions between services.
Bound retained data
If only totals are needed, sum each current group and retain the total. Materializing every group retains all rows; use that only when the input fits the available memory. The lists here contain immutable tuples, so sharing individual rows is harmless. A list copy would still share mutable row objects.
The groupby documentation specifies consecutive groups and their shared source iterator. Sorting a finite input by the same key is one way to bring equal keys together; it is unnecessary when the source already has that order.
Clean up
Remove the saved file and empty scratch directory. If the directory is not empty, inspect its contents before removing anything else.
rm -- group_snapshots.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dirAcknowledgements
This post was created with assistance from AI.