List - dict comprehensions and generators

What are comprehensions?

Definition List comprehension

A list comprehension is a syntactic construct that builds a list by applying an expression to each item in an iterable, optionally filtering items with a condition.

Syntax: [expression for item in iterable if condition] The if clause is optional. This replaces:

result = []
for item in iterable:
   if condition:
       result.append(expression)

WHY comprehensions exist: Loop boilerplate obscures intent. Compare:

# Verbose loop
squares = []
for x in range(10):
    squares.append(x**2)
 
# Comprehension: intent clear in one line
squares = [x**2 for x in range(10)]

The second version is declarative—you state what you want (squared numbers) without the how (initialize list, append in loop). Python's parser can also optimize comprehensions better than manual loops.

Worked example Building a dictionary from two lists

Task: Pair up keys = ['a', 'b', 'c'] and values = [1, 2, 3] into {'a':1, 'b':2, 'c':3}.

keys = ['a', 'b', 'c']
values = [1, 2, 3]
# Dict comprehension
mapping = {k: v for k, v in zip(keys, values)}
# Result: {'a': 1, 'b': 2, 'c': 3}

Why this step?
zip(keys, values) produces tuples ('a',1), ('b',2), ('c',3). Unpacking k, v in the comprehension assigns each tuple's elements. The {k: v ...} syntax builds the dict.

Worked example Filtering with conditions

Task: Extract even squares from 0 to 9.

even_squares = [x**2 for x in range(10) if x % 2 == 0]
# Result: [0, 4, 16, 36, 64]

Why this step?
The if x % 2 == 0 filters before computing x**2. Only values passing the condition enter the output. Order: iterate → filter → transform → collect.

Worked example Nested comprehensions (flattening)

Task: Flatten matrix = [[1,2], [3,4], [5,6]] into [1,2,3,4,5,6].

matrix = [[1,2], [3,4], [5,6]]
flat = [item for row in matrix for item in row]
# Result: [1, 2, 3, 4, 5, 6]

Why this step?
Read left-to-right like nested loops: for row in matrix (outer), then for item in row (inner). Each item is appended. This mirrors:

flat = []
for row in matrix:
   for item in row:
       flat.append(item)

What are generators?

WHY generators: A list [x**2 for x in range(10**9)] allocates space for 1 billion integers (~4-8 GB). A generator (x**2 for x in range(10**9)) allocates ~200 bytes for the iterator object. Values are computed on the fly.

Worked example Generator expression

gen = (x**2 for x in range(5))
print(list(gen))  # [0, 1, 4, 9, 16]
print(list(gen))  # [] — exhausted!

Why this step?
Calling list(gen) forces evaluation: the generator yields0², 1², .., 4² sequentially, collected into a list. After exhaustion, the generator's internal state is at "end", so the second call yields nothing.

Worked example Generator function

def fibonacci(n):
   a, b = 0, 1
   for _ in range(n):
       yield a
       a, b = b, a + b
for num in fibonacci(7):
   print(num, end=' ')
# Output: 0 1 1 2 3 5 8

Why this step?
yield a pauses execution and returns a to the caller. When loop requests the next value, execution resumes right after the yield. The a, b = b, a+b updates state between yields. This is stateful iteration without storing all Fibonacci numbers.

Worked example Chaining generators (pipeline pattern)

Task: Square numbers, filter evens, take first 3.

def squares(n):
   for x in range(n):
       yield x**2
def evens(iterable):
   for item in iterable:
       if item % 2 == 0:
           yield item
result = list(evens(squares(10)))[:3]
# Result: [0, 4, 16]

Why this step?
squares(10) is a generator yielding 0,1,4,9,... evens(...) wraps it, yielding only even values. No intermediate list is created—values flow through the pipeline lazily. Taking [:3] stops iteration early (only computes up to 16).

Comprehensions vs. generators

Feature	List comprehension `[...]`	Generator expression `(...)`
Memory	Stores all items	Stores iterator state only
Speed (first access)	Slower (computes all)	Faster (computes one)
Speed (full iteration)	Faster (data in cache)	Slightly slower (re-compute)
Reusability	Iterate many times	Iterate once
Use case	Small datasets, multiple passes	Large datasets, single pass

Derivation of when to use each:

Let $T_c$ = time to compute one item, $n$ = number of items, $k$ = number of iterations over the collection.

List: Cost = $n \cdot T_c$ (upfront) + $O(1)$ per access × $k$ passes → $(n \cdot T_c + n \cdot k)$
Generator: Cost = $T_c$ per access × $n$ items × $k$ passes → $(n \cdot T_c \cdot k)$

If $k=1$ , generator wins (factor of $k$ smaller). If $k > 1$ and $n$ is small, list wins (avoid recomputation).

Common mistake Using

[] when memory is tight Wrong idea: "I'll just build the list—it's simpler."

Why it feels right: Comprehensions are readable and familiar. Lists let you index and iterate multiple times.

The fix: If the dataset is large and you only need one pass, use a generator expression. Example:

# BAD: loads 10 million records into memory
data = [process(line) for line in open('huge.csv')]
# GOOD: streams one record at a time
data = (process(line) for line in open('huge.csv'))
for record in data:
   train_model(record)  # process and discard

Steel-man: The list version lets you len(data) or access data[100] directly. But if you don't need those, you're paying a memory cost for unused features.

Common mistake Modifying a list while iterating with comprehension

Wrong idea: [x for x in my_list if not my_list.remove(x)]

Why it feels right: "I'll filter by side effect inside the comprehension."

The fix: Comprehensions should be pure expressions. Removing items during iteration causes skipped elements (iterator invalidation). Correct approach:

# Create a new filtered list
my_list = [x for x in my_list if condition(x)]

Or mutate explicitly:

my_list[:] = [x for x in my_list if condition(x)]  # in-place replacement

Recall Explain to a 12-year-old

Imagine you have a toy factory. A list comprehension is like making all the toys first, then stacking them on a shelf. You can grab any toy anytime, but you need a big shelf.

A generator is like having a magic box. When you ask "give me a toy," it builds one toy right then and hands it to you. If you want another, you ask again and it builds the next one. The box is tiny, but you can't go backwards—once you've taken a toy, it's gone from the box.

Dict comprehension is like making a phone book: you pair up names (keys) and phone numbers (values) in one step instead of writing them one by one.

For homework problems with 10 toys, build them all (list). For Santa's workshop making a billion toys, use the magic box (generator)—you'd never fit a billion toys on a shelf!

Connections

1.4.01-Basic-Python-syntaxand-data-structures — comprehensions build on lists, dicts, sets
1.4.02-NumPy-arrays-and-vectorization — NumPy has its own "vectorized" iteration (broadcasting) that's faster than comprehensions for numeric data
1.4.04-File-IO-and-data-loading — generators crucial for streaming large CSV/JSON files
1.5.01-Batch-processing-and-mini-batches — ML data loaders use generators to yield batches lazily
2.3.02-Iterator-protocol — generators implement the iterator protocol under the hood

#flashcards/ai-ml

What is a list comprehension? :: A syntactic construct [expr for item in iterable if condition] that builds a list by applying an expression to each item, optionally filtering. Replaces explicit for-loop-append patterns with a declarative one-liner.

What is a generator?

A function using yield or an expression (expr for ...) that produces values lazily (one at a time, on demand) instead of storing all values in memory. Uses O(1) memory regardless of sequence length.

Why use a generator over a list comprehension?

When working with large datasets and only need one pass—generators use constant memory (iterator state only) instead of allocating space for all items. Tradeoff: can't reuse or index into a generator.

What does yield do in a generator function?

Pauses execution, returns a value to the caller, and saves the function's state. When the next value is requested, execution resumes right after the yield. Enables stateful iteration without storing all results.

What is the memory cost of [x**2 for x in range(10**9)]?

Approximately 4-8 GB (1 billion integers × ~28 bytes per Python int object). A generator (x**2 for x in range(10**9)) costs ~200 bytes (iterator overhead only).

How do you flatten a nested list with comprehension?

[item for row in matrix for item in row]. Read left-to-right like nested loops: outer for row in matrix, then inner for item in row. Each item appends to result.

Can you iterate a generator multiple times?

No. Generators are consumed after one iteration. Calling list(gen) twice yields values the first time, then an empty list the second time (generator is exhausted).

When is a list comprehension faster than a generator?

When you need multiple passes over the data and the dataset fits in memory. List pays computation cost once upfront; generator recomputes on every access. For

k

passes over

n

items, list cost is

n \cdot T_c + n \cdot k

vs generator

n \cdot T_c \cdot k

What is a dict comprehension syntax?

{key_expr: value_expr for item in iterable if condition}. Example: {x: x**2 for x in range(5)} produces {0:0, 1:1, 2:4, 3:9, 4:16}.

Figure — List - dict comprehensions and generators

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Comprehensions aur generators Python ki do powerful tools hain jo apka code clean aur memory-efficient banate hain. Samjho ki apko ek list banana hai jisme har number ka square ho—traditional tarike se ap ek empty list banaoge, for loop chalaoge, aur har iteration mein .append() karoge. List comprehension isko ek line mein compress kar deta hai: [x**2 for x in range(10)]. Ye declarative style hai—aap bata rahe ho kya chahiye, kaise banana hai wo Python khud handle karta hai.

Ab socho ki aapke pas 1 crore records ka dataset hai.Agar aap list comprehension use karoge, to sabhi 1 crore items RAM mein store ho jayenge (GBs of memory!). Generator yahaan hero ban kar ata hai. Generator expression (x**2 for x in range(10**7)) sirf ek iterator object create karta hai jo values on-demand produce karta hai—jab aap next() call karte ho tabek value compute hoti hai. Pora data memoryein nahi baith-ta, to aap badi files bhi efficiently process kar sakte ho. ML mein jab aap large datasets load karte ho batches mein, generators hi use hote hain behind-the-scenes.

Ek aur powerful chez hai dict comprehensions—suppose aapke pas do lists hain,ek keys ka aur ek values ka, to {k:v for k,v in zip(keys, vals)} ek dictionary bana dega single line mein. Ye sab techniques Active Recall ke liye perfect hain: jab aap practice karo to khud se poocho "memory tradeoff kya hai?" ya "generator ko dubara iterate kar sakte hain kya?" (Answer: nahi, woh exhaust ho jata hai ek baar ke bad). Aise savaal apki understanding ko deep banate hain aur ML engineering interviews mein bhi directly ate hain.

List - dict comprehensions and generators

What are comprehensions?

What are generators?

Comprehensions vs. generators

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Python & Scientific Computing

Connections