Python Basics: Producing Values Gradually with Generators

A generator produces values as a caller asks for them. Instead of building a complete result list, a function can pause at yield and resume later. This is useful for processing a stream or connecting small transformations.

These three standalone examples were tested on Linux with Python 3.12.14. They require no third-party packages. Assertions check the demonstrated results; run the commands normally, without Python’s -O option.

Prepare a scratch directory

You need Bash, a text editor, Python 3 on your PATH, and the mktemp, rm, and rmdir utilities. Keep using the same terminal. Save each named source file as UTF-8. The setup stops the shell if creating or entering the scratch directory fails.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

1. Watch execution pause and resume

Save this as trace_generator.py:

def ticket_numbers():
    print("Preparing first ticket")
    yield 101
    print("Preparing second ticket")
    yield 102


tickets = ticket_numbers()
print("Generator created")
first = next(tickets)
assert first == 101
print(f"First: {first}")
remaining = list(tickets)
assert remaining == [102]
print(f"Remaining: {remaining}")
assert list(tickets) == []
print("Generator exhausted")

Run it:

python3 trace_generator.py

Output:

Generator created
Preparing first ticket
First: 101
Preparing second ticket
Remaining: [102]
Generator exhausted

Calling the function creates a generator without running its body. Each request advances it to the next yield. Local execution state is preserved between requests. Once exhausted, this generator stays exhausted; call the function again for a new one.

2. Connect a filtering generator to an aggregation

Save this as sum_readings.py:

def valid_readings(lines):
    for line in lines:
        text = line.strip()
        if not text or text.startswith("#"):
            continue
        yield int(text)


lines = ["# sample readings", "12", "", " 8 ", "5"]
readings = valid_readings(lines)
total = sum(readings)
assert total == 25
assert list(readings) == []
print(f"Total: {total}")
print("No intermediate result list")

Run it:

python3 sum_readings.py

Output:

Total: 25
No intermediate result list

sum consumes values one at a time. The input here is a tiny list so the example is self-contained, but the function accepts other iterables too. Invalid numeric text still raises ValueError when consumption reaches it; lazy evaluation postpones work, including errors.

3. Delegate values with yield from

Save this as combine_batches.py:

def combine_batches(batches):
    for batch in batches:
        yield from batch


batches = [("draft", "review"), (), ("publish",)]
combined = combine_batches(batches)
first = next(combined)
rest = list(combined)
assert first == "draft"
assert rest == ["review", "publish"]
print(f"First: {first}")
print(f"Rest: {rest}")

Run it:

python3 combine_batches.py

Output:

First: draft
Rest: ['review', 'publish']

yield from forwards the values from each batch to the caller. The empty batch produces no values. This example flattens one level and does not pre-build a combined list. Converting the remainder with list is a deliberate choice for displaying this small result.

When to use this

Choose a list when callers need repeated traversal or random indexing. Choose a generator for one-pass processing where delaying work is useful. A generator can still retain its input and local objects, so it is not a promise of constant memory for every program. Read the official generator tutorial.

Clean up

In the same terminal, remove only the three example files, return to your original directory, and remove the empty scratch directory:

rm -- trace_generator.py sum_readings.py combine_batches.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

Acknowledgements

This post was created with assistance from AI.