Generator expressions — memory efficiency

WHAT is a generator expression?

list_comp = [x*x for x in range(1_000_000)]   # builds 1,000,000 ints NOW
gen_expr  = (x*x for x in range(1_000_000))   # builds NOTHING yet

list_comp is a real list holding a million numbers.
gen_expr is a tiny generator object that remembers how to produce them.

WHY does this save memory?

So the memory cost is:

Object	Memory used	Grows with $n$ ?
List of $n$ items	stores all $n$ items	Yes — $O(n)$
Generator	stores 1 item + the recipe	No — $O(1)$

HOW do you actually use it?

You consume it by iterating — for, sum(), next(), any(), etc. Each consumes values once.

# next() pulls ONE value
g = (x*x for x in range(5))
next(g)   # 0
next(g)   # 1   <- state advanced, 0 is gone
 
# A function that takes an iterable consumes the whole generator
total = sum(x*x for x in range(1_000_000))   # parentheses optional as sole arg

Common mistake Steel-man: "I'll just reuse the generator twice"

Why it feels right: lists can be looped over again and again, so you assume generators can too. What actually happens: a generator is exhausted after one full pass — it yields nothing the second time.

g = (x for x in range(3))
list(g)   # [0, 1, 2]
list(g)   # []   <-- empty! already consumed

Fix: rebuild it (g = (...) again), or if you genuinely need it twice, use a list.

Figure — Generator expressions — memory efficiency

Worked Examples

Worked example Example 1 — Summing squares without a million-element list

total = sum(x*x for x in range(1_000_000))

Why this step? sum only needs each square momentarily to add it to a running total. The generator hands them over one at a time, so peak memory stays tiny while a list would allocate ~8 MB+.

Worked example Example 2 — Filtering a huge file lazily

with open("huge.log") as f:
    error_lines = (line for line in f if "ERROR" in line)
    first = next(error_lines)   # reads only until the FIRST error

Why this step? Because of laziness, Python stops reading the file the moment it finds the first match — it never loads the whole file. A list comprehension would scan the entire file first.

Worked example Example 3 — Forecast-then-Verify

Predict the output before reading on:

g = (n for n in range(4) if n % 2 == 0)
print(list(g))
print(sum(g))

Forecast: first line [0, 2], second line 0 (NOT 2). Verify / Why? list(g) consumes the generator fully → [0, 2]. Now g is exhausted, so sum(g) sums nothing → 0. The trap is assuming g resets.

Recall Feynman: explain to a 12-year-old

Imagine you want to eat 100 sandwiches. List way: make all 100 first, pile them on the table (huge mess, lots of space). Generator way: the kitchen makes the next sandwich only when you finish the last one. You're never holding more than one — so you need almost no table space, no matter if it's 100 or a million sandwiches. The catch: once you've eaten them all, they're gone — you can't go back and eat the same ones again.

Active Recall

What punctuation distinguishes a generator expression from a list comprehension?

Parentheses () instead of square brackets [].

What is the memory complexity of a generator expression vs a list comprehension over n items?

Generator is O(1) (one item at a time); list is O(n) (all items stored).

Why does (x*x for x in range(n)) use almost no memory?

It stores only the recipe/state and yields each square on demand, never building the full sequence.

What happens when you iterate a generator a second time?

Nothing is produced — it is exhausted after one full pass.

When summing 1,000,000 squares, why prefer sum(x*x for x in range(10**6)) over a list comprehension?

The generator avoids allocating a million-element list; peak memory stays tiny.

Can you index a generator like g[0]?

No — generators are not subscriptable; you must use next() or iterate.

If a generator expression is the sole argument to a function, what can you drop?

The extra parentheses, e.g. sum(x*x for x in data).

Connections

List comprehensions — same syntax, eager evaluation
Iterators and the iterator protocol — __iter__ / __next__ underpin generators
Generator functions and yield — the def/yield cousin of generator expressions
Lazy evaluation — the general principle of "compute when needed"
Big-O space complexity — why O(1) vs O(n) matters
Memory management in Python — heap allocation of lists

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho yaar, generator expression ka funda simple hai: list comprehension [x for x in ...] saari values ek saath bana ke memory mein rakh deta hai, jabki generator expression (x for x in ...) — sirf brackets badalne se — values ko on demand, ek-ek karke deta hai. Matlab agar tumhe ek lakh squares chahiye, list poore ek lakh numbers memory mein store karegi (heavy!), par generator sirf "recipe" rakhta hai aur jab tum for ya sum se maangte ho tabhi next value banata hai. Isse memory O(n) se O(1) ho jaati hai — chahe 10 items ho ya 1 crore, generator ki memory almost constant rehti hai.

Yeh kyun important hai? Bade data ke saath kaam karte time, jaise badi files padhna ya millions of records process karna, list approach se RAM full ho sakti hai aur program crash. Generator se tum lazy chalte ho — jitna chahiye utna hi banao. Isliye sum(x*x for x in range(10**6)) likhna best practice hai, beech mein faltu ki badi list mat banao.

Ek important trap yaad rakhna: generator ek hi baar chalta hai. Ek baar list(g) ya for se consume kar liya, to dobara woh khaali ho jaata hai — second baar kuch nahi dega. Agar dobara chahiye to phir se generator banao, ya seedha list use karo. Mnemonic: PARENS = PATIENCE — round brackets Python ko patient banate hain (ek-ek value), square brackets greedy (sab ek saath).

Generator expressions — memory efficiency

WHAT is a generator expression?

WHY does this save memory?

HOW do you actually use it?

Worked Examples

Active Recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Introduction to Programming (Python)

Connections