Generator expressions — memory efficiency: Worked examples

You already know the one-line idea from the parent note: a generator expression produces items one at a time, on demand. This page is a drill. We take that idea and hammer it against every kind of situation it can meet — normal cases, empty cases, infinite cases, the "consumed twice" trap, a real-world file problem, and an exam-style twist.

Before we begin, one promise: every symbol and every function that appears (sum, next, any, zip, itertools.islice, memory in bytes) will be explained the first time it shows up. If you have never typed Python in your life, start at line one and you will still follow.

The scenario matrix

Think of a generator expression as a machine with an input (some iterable), a rule (the expr), an optional filter (if cond), and a consumer (whoever pulls the values out). Each of these four parts can be normal, empty, degenerate, or pushed to a limit. The table below lists every case-class this topic can throw at you. Every cell gets a worked example below.

#	Case class	What is special about it	Example
A	Normal, fully consumed	ordinary rule, ordinary consumer	Ex 1
B	Filter removes some	`if cond` drops elements	Ex 2
C	Filter removes ALL (degenerate)	generator yields nothing	Ex 3
D	Empty input (degenerate)	iterable has zero items	Ex 3
E	Partial consumption	pull only a few with `next`	Ex 4
F	Double consumption (the trap)	exhausted after one pass	Ex 5
G	Infinite / limiting	input never ends; must slice	Ex 6
H	Real-world word problem	huge file, memory matters	Ex 7
I	Exam twist	nested / lazy-capture gotcha	Ex 8

Two supporting pictures first, then the examples.

Figure — Generator expressions — memory efficiency

The figure above is the mental model for memory (case classes A–H all live inside it). The list builds every box up front; the generator holds exactly one box and remembers the recipe. Look at the flat green line — that is $O (1)$ , the meaning of "memory does not grow with $n$ ". ( $O (1)$ and $O (n)$ are explained in Big-O space complexity; here just read them as "constant" vs "grows in proportion to $n$ ".)

The second figure is the mental model for consumption: a generator has a pointer that only ever moves forward. That single arrow explains cases E, F and G at once — you can stop early (E), you can run out (F), and you can walk forever (G) — but you can never rewind.

The worked examples

Worked example Example 1 — Case A: normal, fully consumed

total = sum(x*x for x in range(5))
print(total)

Here range(5) is the input 0,1,2,3,4; the rule is x*x (square it); sum(...) is the consumer that adds everything up.

Forecast: what integer prints?

List the squares. range(5) gives 0,1,2,3,4, so the squares are 0,1,4,9,16. Why this step? We must know what values flow out before we can add them. The generator produces them left-to-right, one per pull.
Add them as they arrive. sum keeps a running total: 0 → 1 → 5 → 14 → 30. Why this step? sum never stores the list — it only needs each square for the instant it adds it. That is exactly the $O (1)$ behaviour (flat green line in the first figure).

Verify: $0 + 1 + 4 + 9 + 16 = 30$ . Also a formula check: sum of first $n - 1$ squares $= \frac{( n - 1 ) n ( 2 n - 1 )}{6}$ with $n = 5$ gives $\frac{4 \cdot 5 \cdot 9}{6} = 30$ . ✅

Worked example Example 2 — Case B: a filter removes some elements

evens_cubed = list(n**3 for n in range(6) if n % 2 == 0)
print(evens_cubed)

n % 2 == 0 is the filter: % is the remainder operator, so n % 2 is 0 for even numbers. n**3 is the rule (cube). list(...) is the consumer that does collect everything into a real list — we use it here only to see all the output.

Forecast: which list prints?

Apply the filter first. From 0,1,2,3,4,5 keep only evens: 0,2,4. Why this step? The if cond runs before the rule. A value that fails the filter never even reaches n**3.
Apply the rule to survivors. 0**3=0, 2**3=8, 4**3=64. Why this step? Only the kept values get cubed — this ordering (filter → transform) is the whole reason the syntax reads expr for item ... if cond.

Verify: [0, 8, 64]. Sanity: three inputs survived, three outputs — filter did not change count of survivors, only which. ✅

Worked example Example 3 — Cases C & D: filter removes all, and empty input

a = list(x for x in range(5) if x > 100)   # filter kills everything
b = list(x for x in [] if True)            # empty input
print(a, b)

This is the degenerate corner: in a every value fails the filter; in b there is nothing to filter at all. [] is an empty list.

Forecast: what do a and b look like?

Trace a. No number in 0..4 is > 100, so every value is rejected. The generator yields nothing. Why this step? A generator that yields nothing is still perfectly valid — it is not an error, it is just empty. This is the "filter removes all" edge.
Trace b. There is no item to start with, so the loop body never runs. Why this step? Empty input is the ultimate degenerate case. Notice both paths land in the same place: an empty result.

Verify: [] []. Both empty, no exception raised. Confirms generators degrade gracefully — never a crash on empty. ✅

Worked example Example 4 — Case E: partial consumption with

next

g = (10 * k for k in range(1, 100))
print(next(g))
print(next(g))
print(next(g))

next(g) pulls exactly one value and advances the internal pointer (the forward arrow in the second figure). range(1, 100) starts at 1, so k = 1,2,3,....

Forecast: three numbers print — which?

First pull. k=1 → 10*1 = 10. Pointer now sits after 1. Why this step? next computes just one value; the other 98 are never made. This is the essence of lazy evaluation (Lazy evaluation) — work is deferred until demanded.
Second and third pulls. k=2 → 20, k=3 → 30. Why this step? Each next resumes where the last one stopped — the generator remembers its state. You paid for 3 values, not 99.

Verify: 10, 20, 30. The remaining 96 values (40..990) were never computed — peak memory held one number. ✅

Worked example Example 5 — Case F: the double-consumption trap

g = (n for n in range(4) if n % 2 == 0)
print(list(g))   # first pass
print(sum(g))    # second pass on the SAME g

This is the classic trap. Predict carefully — most people get the second line wrong.

Forecast: guess both lines before reading.

First pass builds the survivors. list(g) walks the whole generator: evens in 0..3 are 0,2 → [0, 2]. Why this step? list drives the pointer to the very end. After this, the forward arrow is past the last element — there is nowhere left to go.
Second pass sees nothing. g is now exhausted. sum(g) iterates an already-finished generator, so it sums zero items → 0. Why this step? A generator cannot rewind (second figure: the arrow only moves forward). The trap is assuming g resets like a list would.

Verify: prints [0, 2] then 0 (not 2). If you truly need both, either rebuild g or store list(g) once and reuse the list. ✅

Worked example Example 6 — Case G: an infinite generator, sliced to a limit

from itertools import count, islice
nats = (m*m for m in count(1))          # 1, 4, 9, 16, ... forever
first_four = list(islice(nats, 4))
print(first_four)

count(1) is an infinite counter 1,2,3,... that never stops. If you did list(nats) your program would run out of memory. islice(nats, 4) means "take the first 4, then stop pulling."

Forecast: which four numbers?

Why a generator is required here, not a list. You literally cannot build a list of all squares — there are infinitely many. Only a lazy iterator can represent an endless sequence without materialising it. Why this tool? This is the case where laziness is not an optimisation but a necessity. A list comprehension would hang forever.
Slice the first four. m=1,2,3,4 → 1,4,9,16. islice stops the moment it has 4, so count never advances to 5. Why this step? The consumer controls how far the pointer walks. Stop demanding, and the infinite source simply pauses — no work wasted.

Verify: [1, 4, 9, 16]. Check: $4^{2} = 16$ , and exactly four elements returned. ✅

Worked example Example 7 — Case H: real-world word problem (huge log file)

Problem. A server log huge.log is 10 GB. You must find whether any line contains "FATAL", without loading the file into RAM.

with open("huge.log") as f:
    found = any("FATAL" in line for line in f)
print(found)

any(...) returns True as soon as it meets one truthy item, else False. Iterating a file object f yields it line by line, lazily.

Forecast: suppose the 3rd line is the first "FATAL". How many lines get read?

Set up the generator over lines. (... for line in f) reads one line at a time — the file is never fully loaded (only one line + the recipe live in memory, the $O (1)$ green line). Why this step? A 10 GB list of lines would blow up RAM. The generator keeps peak memory at roughly one line.
any short-circuits. It pulls line 1 (no match), line 2 (no match), line 3 (match!) and immediately returns True, never touching lines 4…end. Why this step? any + lazy source = stop at the first hit. This combines partial consumption (case E) with a real workload.

Verify (with a stand-in list to make it checkable):

lines = ["ok\n", "warn\n", "FATAL boom\n", "more\n"]
found = any("FATAL" in ln for ln in lines)   # True

found is True; only 3 of 4 lines were inspected. Units sanity: memory scales with one line, not file size. ✅

Worked example Example 8 — Case I: the exam twist (lazy variable capture)

funcs = [lambda: i for i in range(3)]        # list of functions
print([f() for f in funcs])

A lambda: i is a tiny anonymous function returning i. This is the twist exam-writers love. It is not itself a generator expression, but it exposes the same "evaluated later, not now" principle at the heart of laziness — so it belongs in the matrix's tricky corner.

Forecast: you will want to answer [0, 1, 2]. Resist. What actually prints?

When is i read? Each lambda does not capture i's value at creation; it captures the variable i, and reads it only when called. Why this step? This "read the value at call time" is exactly the deferred-computation idea generators embody. The trap is assuming the value is frozen when the lambda is built.
Call them after the loop ends. By the time f() runs, the loop has finished and i has its final value 2 for all three lambdas. Why this step? All three share one variable i; the loop leaves it at 2. Deferred read → all see 2.

Verify: prints [2, 2, 2], not [0, 1, 2]. Fix to get [0,1,2]: bind eagerly with a default arg, lambda i=i: i. ✅

Active Recall

Connections

Generator expressions — memory efficiency — the parent idea this page drills
Lazy evaluation — "computed on demand" powers Examples 4, 6, 8
Iterators and the iterator protocol — next() and the forward-only pointer
Generator functions and yield — the def/yield cousin, same laziness
List comprehensions — the eager alternative used in Examples 2, 8
Big-O space complexity — the $O (1)$ vs $O (n)$ shown in the first figure
Memory management in Python — why the huge-file case (Ex 7) matters