Python Basics: Combining and Batching Iterators

Python’s itertools module helps process values a little at a time. Try combining groups, taking a slice, and splitting work into small batches.

Use Python 3.12 or newer on Linux; batched() was added in 3.12. These independent examples use only in-memory data.

Combine groups without copying them first

Save as demo_itertools_1.py:

from itertools import chain

groups = [["api", "worker"], [], ["cache"]]
services = chain.from_iterable(groups)
print(next(services))
print(list(services))
print(list(services))

Run it from the directory where you saved it:

python3 demo_itertools_1.py

Output:

api
['worker', 'cache']
[]

chain.from_iterable() visits each inner iterable in order. An iterator is consumed as you read it; create a new one to start over. list() is convenient for a small demonstration, but collects all remaining values in memory.

Take a slice of an iterator

Save as demo_itertools_2.py:

from itertools import count, islice

numbers = count(10)
print(list(islice(numbers, 1, 6, 2)))
print(next(numbers))

Run it from the directory where you saved it:

python3 demo_itertools_2.py

Output:

[11, 13, 15]
16

islice() uses zero-based positions: start included, stop excluded. It consumes skipped values too. Fully reading this slice advances the source through position 5. Negative indexes and a nonpositive step are unsupported.

count() has no natural end. Always bound it with something like islice(); converting the unbounded iterator directly to a list would keep consuming memory.

Process small batches

Save as demo_itertools_3.py:

from itertools import batched

jobs = (f"job-{number}" for number in range(1, 6))
for batch in batched(jobs, 2):
    print(batch)
print("Empty input:", list(batched([], 2)))

try:
    list(batched(["job-1"], 0))
except ValueError:
    print("Batch size must be at least 1")

Run it from the directory where you saved it:

python3 demo_itertools_3.py

Output:

('job-1', 'job-2')
('job-3', 'job-4')
('job-5',)
Empty input: []
Batch size must be at least 1

Each batch is a tuple. The last can be smaller, so downstream code must accept an incomplete batch. Keep the loop when processing large input rather than collecting every batch. Python 3.12 does not support the later strict keyword.

References

• Python itertools documentation

Acknowledgements

This post was created with assistance from AI.