Processing massive datasets can crash your computer if you load everything into memory at once.
That's where Generators and Iterators shine: they let you process data one item at a time, "on the fly," making your code incredibly memory-efficient.
1. Memory Limits (The Problem)
Creating a list like
my_list = [i*i for i in range(1_000_000)] stores all million squares in your RAM simultaneously. This is fine for small data, but unsustainable for massive files or infinite streams.2. Generators & Iterators (The Solution )
🟢 Iterator: An object that represents a stream of data. It only gives you the
next() item when you ask for it, raising StopIteration when the data runs out.🟢 Generator: The simplest way to create an iterator. It's a function that uses the
yield keyword instead of return.*
return: Ends the function and sends back a final result.*
yield: Pauses the function, sends back a value, and "hibernates" until you ask for the next one.3. Lazy Evaluation
This is the "magic" of generators. They don't calculate any values until you specifically ask for them. This is called Lazy Evaluation.
🐍 Concise Code Example
# A generator function
def countdown(n):
print("Starting countdown...")
while n > 0:
yield n # Pauses here and returns n
n -= 1
# 1. Create the generator object (No code inside the function runs yet!)
timer = countdown(3)
# 2. Get values one by one using next()
print(next(timer)) # Output: Starting countdown... \n 3
print(next(timer)) # Output: 2
# 3. Generators are most commonly used in loops
for number in countdown(2):
print(number)
# Output: 2, 1
4. Why Use Them?
📌 Low Memory Footprint: You only ever have one item in memory at a time.
📌 Infinite Sequences: You can create a generator that runs forever (e.g., a sensor data stream) without crashing your app.
📌 Pipeline Processing: You can "chain" generators together to transform data in stages without creating massive intermediate lists.
🎯 Today's Goal (What you should do)
✔️ Understand the difference between
yield (pause) and return (stop)✔️ Recognize when to use a generator to prevent memory overflow
✔️ Master the use of
next() and for loops to consume iterators✔️ Learn the concept of "Lazy Evaluation" for high-performance data processing
👉 Generators turn your code from a memory-hungry "buffet" into an efficient "made-to-order" kitchen!