Generators — yield, generator functions, send(), next()

WHAT is a generator?

def count_up(n):
    i = 0
    while i < n:
        yield i        # pause here, hand out i
        i += 1         # resume here next time
 
g = count_up(3)        # NOTHING runs yet; g is a generator object
print(next(g))   # 0   <- runs until first yield, pauses
print(next(g))   # 1   <- resumes after yield, runs to next yield
print(next(g))   # 2
print(next(g))   # raises StopIteration  (loop ended)

HOW does pausing actually work? (derive it from scratch)

You can build the concept of a generator with a plain class, which proves there's no magic:

class CountUp:
    def __init__(self, n):
        self.n = n
        self.i = 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.i >= self.n:
            raise StopIteration        # the "end of body"
        val = self.i                   # the "yield value"
        self.i += 1                    # the "resume" bookkeeping
        return val

So a generator is exactly: an object with __iter__ and __next__, but the saving of position is done by the interpreter, not by you.

The protocol: `next()`, `StopIteration`, and `for`

for x in count_up(3):   # calls next() until StopIteration, then stops
    print(x)            # 0, 1, 2

`send()` — talking BACK into a generator

Figure — Generators — yield, generator functions, send(), next()

Worked example A running-average coroutine

def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        x = yield average      # send 'average' OUT, get next 'x' IN
        total += x
        count += 1
        average = total / count
 
g = averager()
next(g)            # PRIME it: run up to first yield (yields None)
print(g.send(10)) # 10.0   -> x became 10
print(g.send(20)) # 15.0
print(g.send(30)) # 20.0

Why prime with next(g) first? Because on the very first run the generator hasn't reached a yield yet, so there is no paused yield expression to receive a value. send(non-None) before priming raises TypeError. Why this step? You can only "answer" a question after the generator has "asked" it (reached yield).

Worked examples

Worked example 1 — Lazy infinite sequence (memory win)

def naturals():
    n = 1
    while True:        # infinite! safe because it's lazy
        yield n
        n += 1
 
def take(gen, k):
    out = []
    for _ in range(k):
        out.append(next(gen))   # pull exactly k values
    return out
 
print(take(naturals(), 5))   # [1, 2, 3, 4, 5]

Why this step? while True would hang a normal function. In a generator it's fine — values only materialize when pulled, so we control how many.

Worked example 2 — Pipelining (chaining generators)

def read_numbers(lines):
    for line in lines:
        yield int(line.strip())
 
def squares(nums):
    for n in nums:
        yield n * n
 
data = ["1\n", "2\n", "3\n"]
pipe = squares(read_numbers(data))   # nothing computed yet
print(list(pipe))   # [1, 4, 9]

Why this step? Each generator pulls one item from the previous only when asked. The whole pipeline streams one element at a time — never loads all data.

Worked example 3 —

yield from (delegation)

def chain(*iterables):
    for it in iterables:
        yield from it        # re-yield every value of it
 
print(list(chain([1,2], (3,4), "ab")))   # [1, 2, 3, 4, 'a', 'b']

Why this step? yield from it replaces a manual for x in it: yield x loop and also correctly forwards send()/exceptions/return values.

Common mistakes (steel-manned)

80/20 — the 20% that gives 80%

Recall Feynman: explain to a 12-year-old

Imagine a story-teller who tells you ONE sentence, then freezes mid-sentence and waits. When you say "next!", they continue from the exact word they stopped at — they never forget where they were. That's a generator: yield is the freeze, next() is your "next!", and send() is you whispering a word that they use to decide what to say next. Because they only speak one sentence at a time, they don't need the whole book in their head — that's why it saves memory.

Flashcards

What makes a function a generator function?

Its body contains the yield keyword.

What does calling a generator function actually do?

Returns a generator object immediately; the body runs only on the first next().

What does next(g) do?

Resumes execution until the next yield (returns that value) or raises StopIteration if the body ends.

Difference between yield and return?

yield pauses and saves state to resume later; return ends the function (in a generator it raises StopIteration).

Why are generators memory-efficient?

They produce values lazily on demand instead of building/storing the whole sequence.

Can you iterate a generator twice?

No — it is single-use; once exhausted it yields nothing.

What does g.send(v) do?

Resumes the generator, making the paused yield expression evaluate to v, and returns the next yielded value.

Why must you prime a generator before send(non-None)?

It hasn't reached any yield yet, so there's no paused yield expression to receive the value; otherwise TypeError.

next(g) is equivalent to which send call?

g.send(None).

What does yield from iterable do?

Yields every value of the sub-iterable and forwards send/exceptions and captures its return value.

How does a for loop know when to stop on a generator?

It catches the StopIteration raised by __next__ and ends silently.

Is a generator its own iterator?

Yes — iter(g) is g, and it has both __iter__ and __next__.

Connections

Iterators and the Iterator Protocol — generators are the easiest way to make one.
List Comprehensions vs Generator Expressions — (x for x in ...) is a generator.
Lazy Evaluation — generators delay work until needed.
Coroutines and async — send() is the foundation of coroutine-style code.
StopIteration and Exceptions — the signal that ends iteration.
Memory and Big Data Streaming — why pipelines use generators.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek normal function ek baar chalता hai, ek value return karता hai, aur khatam. Lekin generator ek aisa function hai jisme yield hota hai — yeh function khud ko pause kar deta hai, ek value tumhe deta hai, aur apni jagah yaad rakhta hai. Jab tum dobara next() bulाte ho, toh wo wahीं se resume hota hai jahan ruka tha. Isliye generator ko lazy kehte hain — values tabhi banti hain jab tum maangte ho, isliye memory bahut kam lagti hai. count_up(10**12) bhi crash nahi karega kyunki wo poori list nahi banata.

Yaad rakho ek important baat: generator function ko call karne se body nahi chalti! g = count_up(3) sirf ek generator object banata hai. Asli code tab chalta hai jab tum pehli baar next(g) karte ho. Aur jab body khatam ho jaati hai toh StopIteration raise hota hai — for loop ise chupke se pakad leta hai, isliye loop khud band ho jaata hai.

send() thoda advanced hai. yield sirf bahar value nahi bhejta — x = yield average me, jab tum g.send(10) karte ho, toh x ki value 10 ban jaati hai. Matlab tum generator ke andar value wapas bhej sakte ho. Lekin pehle generator ko prime karna padta hai — yaani ek next(g) chalाo taaki wo pehle yield tak pahunch jaaye. Bina prime kiye send(non-None) karoge toh TypeError aayega, kyunki abhi koi paused yield hai hi nahi jo value le sake.

Yeh concept isliye important hai kyunki real duniya me bade data files, streams, ya pipelines me hum saara data ek saath memory me nahi laa sakte. Generators se hum ek-ek karke data process karte hain — fast, light, aur clean. Aur send() se aage chalke coroutines aur async programming ki neev padti hai.

Generators — yield, generator functions, send(), next()

WHAT is a generator?

HOW does pausing actually work? (derive it from scratch)

The protocol: `next()`, `StopIteration`, and `for`

`send()` — talking BACK into a generator

Worked examples

Common mistakes (steel-manned)

80/20 — the 20% that gives 80%

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Python Intermediate

Connections