1.4.3 · D5Python & Scientific Computing

Question bank — List - dict comprehensions and generators

1,730 words8 min readBack to topic

Before we start, two things we lean on constantly. First a word:

And second a size shorthand you'll see twice below:

To picture that gap — the single idea driving half this page — look at this:

Figure — List - dict comprehensions and generators

The red line (generator) stays flat no matter how many items you demand; the black line (list) climbs with every item. That flat red line is what looks like.


True or false — justify

A list comprehension always uses less memory than the equivalent loop with .append()
False. Both produce the same full list in memory, so peak memory is essentially identical; the comprehension is just faster and shorter, not lighter.
A generator expression uses less memory than a list comprehension
True — but only because it never builds the full collection. It stores just the cursor state, so its saving is the whole point, not a side effect.
(x**2 for x in range(5)) and [x**2 for x in range(5)] produce the same object
False. Brackets [] build a list (all values now); parentheses () build a generator (values on demand). They print differently and behave differently on a second pass.
You can call len() on a generator
False. A generator has no stored contents to count — it would have to run to the end to know its length, which also consumes it. Only materialized collections like lists support len().
Iterating a list twice gives the same items both times
True. The list is not consumed; each for loop makes a fresh cursor from the start. This is exactly what a generator cannot promise.
A dict comprehension can produce fewer entries than the iterable has items
True. If two items map to the same key, the later one overwrites the earlier — so duplicate keys silently shrink the result.
{x: x**2 for x in [1, -1, 2]} has three entries
True. The keys are 1, -1, and 2 — all distinct — so you get three entries {1: 1, -1: 1, 2: 4}. Note 1 and -1 map to the same value 1, but distinct keys never collide, so nothing is lost.
A generator function must contain a return statement to give back values
False. It uses yield to hand out values; a bare return just ends iteration early. Mixing them, return stops the generator without producing a final value.
Set comprehensions and generator expressions both use braces {}
False. Set comprehensions use {expr for ...}; generator expressions use (expr for ...). Braces mean set (or dict, with a colon), never generator.

Spot the error

[x for x in my_list if not my_list.remove(x)]
Two bugs. list.remove(x) returns None, so not None is always True — the filter keeps everything the loop reaches. Worse, remove deletes the current item and shifts every later element one slot left, but the cursor still advances to the next index — so it now lands on what used to be two slots away, silently skipping the element that slid into the gap. Never mutate what you iterate; build a new list: [x for x in my_list if keep(x)].
gen = (i for i in range(3)); print(list(gen)); print(sum(gen)) — the sum is 0
list(gen) consumed the generator, so sum(gen) iterates an empty, exhausted cursor and returns 0. Rebuild the generator or use a list if you need two passes.
squares = (x**2 for x in range(1000000)); train(squares); test(squares)
train consumes the generator, leaving test nothing. For multiple passes over the same large data, either re-create the generator each time or accept the memory cost of a list.
total = sum([x**2 for x in range(10**8)]) on a memory-limited machine
The brackets force a 100-million-element list into memory before summing. Drop the brackets — sum(x**2 for x in range(10**8)) streams one value at a time in (constant) space.
{k: v for k in keys for v in values} to pair two lists
This is a nested loop, pairing every key with every value and keeping only the last v — it does not zip them. Use {k: v for k, v in zip(keys, values)}.
flat = [item for item in row for row in matrix]
The loop order is reversed — row is used before it is defined. Nested comprehensions read left-to-right like nested loops: [item for row in matrix for item in row].
def f(): yield 1; return 5 — expecting list(f()) to be [1, 5]
The return 5 does not yield 5; it ends the generator (and sets a StopIteration value nobody usually reads). Result is [1].

Why questions

Why does a comprehension usually beat a hand-written append loop in speed?
Because the list-building loop runs in optimized C internals rather than repeated Python-level .append() attribute lookups and calls, so Python's interpreter has less per-iteration overhead.
Why can a generator stream a 10 GB file through 1 MB of RAM?
It never holds all the data at once — it reads, transforms, and hands back one record, then forgets it. Only the current item plus the cursor's tiny state live in memory. See file IO.
Why do we say generators are "lazy"?
They compute nothing until a value is actually requested. Wrapping generators (like evens(squares(n))) builds a pipeline that only advances one step per requested item — perfect for mini-batching.
Why does if condition in a comprehension run before the expression?
The comprehension reads as filter-then-transform: each item is tested, and only survivors reach the expression. So [x**2 for x in r if x % 2 == 0] never squares odd numbers.
Why prefer a generator for a single-pass training loop, but a list for repeated epochs over small data?
Let be the number of times you loop over the whole collection (one pass means ; five training epochs means ). When a generator avoids storing everything for no benefit. When over small data, a list computes each item once and re-serves it cheaply across all passes, avoiding recomputation.
Why is a generator "faster on first access" but "slower on full iteration" than a list?
On first access it computes only one value instead of the whole collection. Over a full pass it recomputes each item live, whereas a pre-built list just reads finished values sitting in cache.
Why does the iterator protocol matter for generators?
A generator is defined by that protocol — next() resumes it, StopIteration ends it. Understanding the protocol explains exactly why a consumed generator yields nothing on a second loop.

Edge cases

What does list(x for x in range(0)) return?
An empty list []. An empty iterable yields nothing, but this is perfectly valid — no error, just zero items.
What does a generator function with zero yield executions produce?
An empty sequence. If the loop inside never reaches a yield (e.g. fibonacci(0)), iterating it gives nothing, cleanly.
What happens if the if condition filters out every item?
You get an empty list, dict, or generator — never an error. Filtering to nothing is a legal, common result.
For {x % 2: x for x in range(5)}, how many entries result?
Two — keys can only be 0 or 1, and each later collision overwrites, so you keep {0: 4, 1: 3}.
Can you nest a generator expression inside a list comprehension?
Yes — e.g. [sum(y for y in row) for row in matrix]. The inner generator stays lazy; the outer comprehension materializes the per-row sums.
What does calling next() on an already-exhausted generator do?
It raises StopIteration (which a for loop silently catches to stop). The generator is done and will never produce more values.
Does a dict comprehension over an empty iterable error out?
No — it produces an empty dict {}. Like all comprehensions, zero iterations is a valid, silent outcome.
Recall One-line survival rule

If you loop once and data is big → generator. If you loop many times or need len/indexing → list. If you mutate what you iterate → you already have a bug.