Noesis
1.2.36 · D5Introduction to Programming (Python)

Question bank — Generator expressions — memory efficiency

1,447 words7 min read↑ Back to topic
🇮🇳 Yeh page Hinglish mein padho →

This bank drills the conceptual traps of generator expressions — not arithmetic. Every item targets a misconception or a boundary case. Related ideas: Lazy evaluation, Iterators and the iterator protocol, List comprehensions, Big-O space complexity.

Before we start, one shared vocabulary reminder so no word is used before it's pinned down:

True or false — justify

A generator expression runs its loop body as soon as you write it.
False. Writing (x*x for x in ...) builds only a generator object; the loop body runs later, one step per value pulled — this is Lazy evaluation.
(x for x in range(3)) and [x for x in range(3)] use the same amount of memory.
False. The list materialises all three items (); the generator stores only its state and the current item ().
A generator expression can be iterated as many times as you like.
False. It is single-use: after one full pass it is exhausted and every later iteration yields nothing.
Putting parentheses around a comprehension always makes a generator.
False-ish — a bare (expr) is just grouping; you need the for inside, e.g. (expr for item in it). (5) is the number 5, not a generator.
A generator with an if filter still visits every element of the source.
True. The filter decides what to yield, but the underlying iterable is still walked item by item as you consume it.
Calling len() on a generator expression tells you how many items it will produce.
False. Generators have no length — Python would have to run the whole thing to count, which defeats laziness, so len(g) raises TypeError.
If you never iterate a generator, its loop body never executes at all.
True. No consumption means no computation — a generator you create and drop does essentially nothing.
Converting a generator to a list with list(g) gives back the same memory savings.
False. list(g) materialises everything, so you're back to memory — you threw the laziness away.

Spot the error

g = (x*x for x in range(5))
print(g[0])

::: Error — generators are not subscriptable. g[0] raises TypeError; you must use next(g) or iterate, because there is no stored element to index into.

g = (x for x in range(3))
a = list(g)
b = list(g)   # expecting [0,1,2] again

::: b is [], not [0,1,2]. The first list(g) exhausted the generator, so the second pass sees nothing — a classic single-pass trap.

total = sum(x for x in range(10) if x % 2)
avg = total / len(x for x in range(10) if x % 2)

::: The len(...) on a generator raises TypeError, and even conceptually it's wrong: the generator has no length. Store the values in a list first if you need both sum and count.

def make():
    return (line for line in open("data.txt"))
gen = make()
# ... later, file already closed elsewhere ...
first = next(gen)

::: The generator is lazy, so the file is read when consumed, not when make() returned. If the file was closed before you call next, you get a "read of closed file" error — laziness moves when side-effects happen.

pairs = (a, b for a in range(2) for b in range(2))

::: Syntax error — (a, b for ...) is ambiguous. To yield tuples you must parenthesise the expression: ((a, b) for a in range(2) for b in range(2)).

nums = (x for x in range(3))
if nums:
    print("has items")

::: The if nums is always truthy — a generator object is truthy even when empty or exhausted. Testing truthiness does not tell you whether it has values; you'd have to try next().

Why questions

Why is a generator's memory regardless of how many values it will yield?
Because it stores only fixed state (a frame and a position pointer) plus the one current item; the "number stored at once" is 1, so the per-element term never accumulates — see Big-O space complexity.
Why can next(error_lines) on a lazy file filter stop after reading only part of the file?
Laziness computes each line only on demand, so the moment the first match is found, next returns and no further lines are read.
Why does sum(x*x for x in range(10**6)) beat the list version on memory but not necessarily on speed?
Memory wins because no million-element list is allocated; speed is similar because the same million squares are still computed one by one — laziness saves space, not work.
Why can you drop the inner parentheses in sum(x*x for x in data)?
When a generator expression is the sole argument to a function call, Python lets the call's own parentheses serve as the generator's, so sum((...)) may be written sum(...).
Why does a generator that filters with if cond sometimes look "slow to start"?
It must keep pulling from the source until a value passes the filter; if matches are rare, next walks many rejected items before yielding one — the work is deferred, not skipped.
Why does storing a generator in a variable not protect its values from being consumed?
The variable holds the same single stateful object; any consumer that advances it (a for, a sum, even a stray list) permanently moves the shared position forward.

Edge cases

What does list(x for x in range(0)) produce?
[] — an empty range yields nothing, so the generator is born already exhausted; this is the zero-length boundary, and it's a valid, error-free result.
What does next(g) do once g is fully exhausted?
It raises StopIteration — the same signal a for loop catches silently to know it should stop.
Can a generator expression be infinite, and is that safe?
Yes, e.g. (x for x in itertools.count()); it's safe only if you never fully materialise it — list() or sum() on it would loop forever, but next() a few times is fine.
What happens if the source iterable changes while the generator is mid-iteration?
Because it reads lazily, later next calls see the current state of the source; mutating the source during iteration can skip or repeat items and is a common bug.
Does a filter that matches nothing, like (x for x in range(5) if x > 100), raise an error?
No — it simply yields no values and behaves like an empty generator; list(...) gives [] and next(...) raises StopIteration.
Is (x for x in []) different from (x for x in range(0)) in behaviour?
No — both wrap an empty source and yield nothing; the source type differs but the exhausted-from-birth behaviour is identical.
Recall One-line summary to carry away

A generator expression is a lazy, single-pass, -memory recipe: it computes on demand, walks its source once, holds one item at a time, and cannot be indexed, measured with len, or replayed.

Connections

  • Lazy evaluation — why "compute on demand" underlies every trap here
  • Iterators and the iterator protocol — the single-pass / StopIteration mechanics
  • List comprehensions — the eager [] cousin that can be reused and indexed
  • Big-O space complexity — the vs distinction
  • Generator functions and yield — the def/yield sibling with the same laziness