Complete Guide to Python Generators and Yield Expressions

Loading massive datasets into list collections can rapidly exhaust system RAM.

Python Generators solve this memory bottleneck by producing elements lazily, computing them one at a time on demand instead of generating the entire series at once.

1. Generator Functions and the yield Keyword

When a function contains the yield statement, it becomes a generator function. It yields control back to the caller while preserving its execution state:

def count_up_to(limit):
    count = 1
    while count <= limit:
        yield count
        count += 1

# Instantiate the lazy generator object
counter = count_up_to(3)

print(next(counter))  # Outputs: 1
print(next(counter))  # Outputs: 2
print(next(counter))  # Outputs: 3

2. Massive Memory Performance Difference

Compare a standard list generator to a generator expression when building a sequence of millions of squared numbers:

import sys

# Standard List Comprehension (Loads everything into memory)
huge_list = [x ** 2 for x in range(1000000)]
print("List Size in Bytes:", sys.getsizeof(huge_list))

# Lazy Generator Expression (Only evaluates on demand)
huge_generator = (x ** 2 for x in range(1000000))
print("Generator Size in Bytes:", sys.getsizeof(huge_generator))

The generator expression uses a fixed minimal size in memory, regardless of how many million elements it handles.