Generator expressions — memory efficiency: Worked examples

1.2.36 · D3 · Coding › Introduction to Programming (Python) › Generator expressions — memory efficiency

Tum parent note se ek-line idea already jaante ho: ek generator expression items ko ek ek karke, demand par produce karta hai. Yeh page ek drill hai. Hum us idea ko lete hain aur use har tarah ki situation se takraate hain — normal cases, empty cases, infinite cases, "consumed twice" trap, ek real-world file problem, aur ek exam-style twist.

Shuru karne se pehle, ek promise: har symbol aur har function jo appear karta hai (sum, next, any, zip, itertools.islice, memory in bytes) — use pehli baar aane par explain kiya jayega. Agar tumne zindagi mein kabhi Python type nahi ki, line one se shuru karo aur phir bhi samajh mein aayega.

The scenario matrix

Generator expression ko ek machine ki tarah socho jisme ek input hota hai (koi iterable), ek rule hota hai (expr), ek optional filter hota hai (if cond), aur ek consumer hota hai (jo bhi values bahar kheenchta hai). In chaaron parts mein se har ek normal, empty, degenerate, ya limit tak pushed ho sakta hai. Neeche diya table har case-class list karta hai jo is topic mein aa sakti hai. Har cell ka ek worked example neeche diya gaya hai.

#	Case class	Isme kya khaas hai	Example
A	Normal, fully consumed	ordinary rule, ordinary consumer	Ex 1
B	Filter removes some	`if cond` elements drop karta hai	Ex 2
C	Filter removes ALL (degenerate)	generator kuch yield nahi karta	Ex 3
D	Empty input (degenerate)	iterable mein zero items hain	Ex 3
E	Partial consumption	`next` se sirf kuch hi pull karo	Ex 4
F	Double consumption (the trap)	ek pass ke baad exhausted ho jaata hai	Ex 5
G	Infinite / limiting	input kabhi khatam nahi hoti; slice karna padta hai	Ex 6
H	Real-world word problem	huge file, memory matters	Ex 7
I	Exam twist	nested / lazy-capture gotcha	Ex 8

Pehle do supporting pictures, phir examples.

Figure — Generator expressions — memory efficiency

Upar wala figure memory ke liye mental model hai (case classes A–H sab iske andar aate hain). List har box pehle se bana leti hai; generator exactly ek box hold karta hai aur recipe yaad rakhta hai. Flat green line dekho — woh $O (1)$ hai, yaani "memory $n$ ke saath nahi badhti". ( $O (1)$ aur $O (n)$ ko Big-O space complexity mein explain kiya gaya hai; yahan inhe sirf "constant" vs " $n$ ke proportion mein badhne wala" padho.)

Doosra figure consumption ke liye mental model hai: ek generator ka ek pointer hota hai jo sirf aage badhta hai. Woh single arrow cases E, F aur G ko ek saath explain karta hai — tum jaldi rok sakte ho (E), tum khatam ho sakte ho (F), aur tum forever chal sakte ho (G) — lekin tum kabhi rewind nahi kar sakte.

The worked examples

Worked example Example 1 — Case A: normal, fully consumed

total = sum(x*x for x in range(5))
print(total)

Yahan range(5) input hai 0,1,2,3,4; rule hai x*x (square karo); sum(...) consumer hai jo sab kuch add karta hai.

Forecast: kaun sa integer print hoga?

Squares list karo. range(5) deta hai 0,1,2,3,4, toh squares hain 0,1,4,9,16. Yeh step kyun? Hume jaanna hai ki kaunsi values flow out hongi tabhi hum unhe add kar sakte hain. Generator inhe left-to-right, ek pull par ek produce karta hai.
Jaise aate hain add karo. sum ek running total rakhta hai: 0 → 1 → 5 → 14 → 30. Yeh step kyun? sum kabhi list store nahi karta — use har square sirf us pal ke liye chahiye jab woh use add karta hai. Yahi $O (1)$ behaviour hai (pehle figure mein flat green line).

Verify: $0 + 1 + 4 + 9 + 16 = 30$ . Formula check bhi: pehle $n - 1$ squares ka sum $= \frac{( n - 1 ) n ( 2 n - 1 )}{6}$ jisme $n = 5$ se $\frac{4 \cdot 5 \cdot 9}{6} = 30$ milta hai. ✅

Worked example Example 2 — Case B: ek filter kuch elements remove karta hai

evens_cubed = list(n**3 for n in range(6) if n % 2 == 0)
print(evens_cubed)

n % 2 == 0 filter hai: % remainder operator hai, toh n % 2 even numbers ke liye 0 hota hai. n**3 rule hai (cube). list(...) consumer hai jo sab kuch ek real list mein collect karta hai — hum ise yahan sirf saara output dekhne ke liye use kar rahe hain.

Forecast: kaun si list print hogi?

Pehle filter lagao. 0,1,2,3,4,5 mein se sirf evens rakho: 0,2,4. Yeh step kyun? if cond rule se pehle run hota hai. Jo value filter fail karti hai woh n**3 tak pahunchi hi nahi.
Survivors par rule lagao. 0**3=0, 2**3=8, 4**3=64. Yeh step kyun? Sirf rakhe gaye values cube hote hain — yeh ordering (filter → transform) hi wajah hai ki syntax expr for item ... if cond jaisi padhti hai.

Verify: [0, 8, 64]. Sanity check: teen inputs survive kiye, teen outputs — filter ne survivors ki count nahi badli, sirf kaunse the woh badla. ✅

Worked example Example 3 — Cases C & D: filter sab remove kar deta hai, aur empty input

a = list(x for x in range(5) if x > 100)   # filter sab kuch khatam kar deta hai
b = list(x for x in [] if True)            # empty input
print(a, b)

Yeh degenerate corner hai: a mein har value filter fail karti hai; b mein filter karne ke liye kuch hai hi nahi. [] ek empty list hai.

Forecast: a aur b kaisi dikhti hain?

a trace karo. 0..4 mein koi bhi number > 100 nahi hai, toh har value reject ho jaati hai. Generator kuch yield nahi karta. Yeh step kyun? Ek generator jo kuch yield nahi karta woh bilkul valid hai — yeh error nahi hai, bas empty hai. Yeh "filter removes all" edge case hai.
b trace karo. Shuru karne ke liye koi item hai hi nahi, toh loop body kabhi run hi nahi hoti. Yeh step kyun? Empty input ultimate degenerate case hai. Notice karo ki dono paths ek hi jagah jaate hain: empty result.

Verify: [] []. Dono empty, koi exception raise nahi hua. Confirm karta hai ki generators gracefully degrade hote hain — kabhi crash nahi empty par. ✅

Worked example Example 4 — Case E:

next se partial consumption

g = (10 * k for k in range(1, 100))
print(next(g))
print(next(g))
print(next(g))

next(g) exactly ek value pull karta hai aur internal pointer ko aage badhata hai (doosre figure mein forward arrow). range(1, 100) 1 se shuru hota hai, toh k = 1,2,3,....

Forecast: teen numbers print hote hain — kaun se?

Pehla pull. k=1 → 10*1 = 10. Pointer ab 1 ke baad hai. Yeh step kyun? next sirf ek value compute karta hai; baaki 98 kabhi bante hi nahi. Yahi lazy evaluation (Lazy evaluation) ka essence hai — kaam tab tak defer hota hai jab tak demand na ho.
Doosra aur teesra pull. k=2 → 20, k=3 → 30. Yeh step kyun? Har next wahan se resume karta hai jahan pichhla ruka tha — generator apni state yaad rakhta hai. Tumne 3 values ki keemat chukaayi, 99 ki nahi.

Verify: 10, 20, 30. Baaki 96 values (40..990) kabhi compute hi nahi huin — peak memory mein sirf ek number tha. ✅

Worked example Example 5 — Case F: double-consumption trap

g = (n for n in range(4) if n % 2 == 0)
print(list(g))   # pehla pass
print(sum(g))    # SAME g par doosra pass

Yeh classic trap hai. Dhyan se predict karo — zyaadatar log doosri line galat karte hain.

Forecast: padhne se pehle dono lines guess karo.

Pehla pass survivors banata hai. list(g) poore generator ko walk karta hai: 0..3 mein evens hain 0,2 → [0, 2]. Yeh step kyun? list pointer ko bilkul end tak drive karta hai. Iske baad, forward arrow last element ke past hai — aage jaane ki koi jagah nahi bachi.
Doosra pass kuch nahi dekhta. g ab exhausted hai. sum(g) ek already-finished generator iterate karta hai, toh woh zero items add karta hai → 0. Yeh step kyun? Generator rewind nahi ho sakta (doosra figure: arrow sirf aage badhta hai). Trap yeh assume karna hai ki g list ki tarah reset ho jaata hai.

Verify: pehle [0, 2] print hota hai phir 0 (not 2). Agar tumhe dono sach mein chahiye, ya toh g rebuild karo ya ek baar list(g) store karo aur list reuse karo. ✅

Worked example Example 6 — Case G: ek infinite generator, limit tak sliced

from itertools import count, islice
nats = (m*m for m in count(1))          # 1, 4, 9, 16, ... forever
first_four = list(islice(nats, 4))
print(first_four)

count(1) ek infinite counter hai 1,2,3,... jo kabhi nahi rukta. Agar tum list(nats) karte toh program memory khatam kar deta. islice(nats, 4) ka matlab hai "pehle 4 lo, phir pulling band karo."

Forecast: kaun se chaar numbers?

Yahan generator kyun zaruri hai, list nahi. Tum literally sab squares ki list nahi bana sakte — infinitely many hain. Sirf ek lazy iterator hi bina materialize kiye ek endless sequence ko represent kar sakta hai. Yeh tool kyun? Yeh woh case hai jahan laziness optimization nahi balki ek zarurat hai. List comprehension forever hang kar jaati.
Pehle chaar slice karo. m=1,2,3,4 → 1,4,9,16. islice jaise hi 4 ho jaate hain ruk jaata hai, toh count kabhi 5 tak nahi badhta. Yeh step kyun? Consumer control karta hai ki pointer kitna aage jaayega. Demand karna band karo, aur infinite source simply pause ho jaata hai — koi kaam waste nahi.

Verify: [1, 4, 9, 16]. Check: $4^{2} = 16$ , aur exactly chaar elements returned. ✅

Worked example Example 7 — Case H: real-world word problem (huge log file)

Problem. Ek server log huge.log 10 GB ka hai. Tumhe pata lagana hai ki kya koi line mein "FATAL" hai, bina file ko RAM mein load kiye.

with open("huge.log") as f:
    found = any("FATAL" in line for line in f)
print(found)

any(...) jaise hi ek truthy item milta hai True return karta hai, warna False. File object f ko iterate karna use line by line, lazily yield karta hai.

Forecast: maano 3rd line pehli "FATAL" hai. Kitni lines read hoti hain?

Lines par generator set karo. (... for line in f) ek ek line padhta hai — file kabhi fully load nahi hoti (memory mein sirf ek line + recipe rehti hai, $O (1)$ green line). Yeh step kyun? 10 GB lines ki list RAM blow up kar deti. Generator peak memory ko roughly ek line par rakhta hai.
any short-circuit karta hai. Woh line 1 pull karta hai (match nahi), line 2 (match nahi), line 3 (match!) aur immediately True return karta hai, lines 4…end kabhi nahi choota. Yeh step kyun? any + lazy source = pehli hit par ruk jao. Yeh partial consumption (case E) ko ek real workload ke saath combine karta hai.

Verify (ek stand-in list ke saath checkable banana ke liye):

lines = ["ok\n", "warn\n", "FATAL boom\n", "more\n"]
found = any("FATAL" in ln for ln in lines)   # True

found hai True; 4 mein se sirf 3 lines inspect huin. Units sanity: memory ek line ke saath scale hoti hai, file size ke saath nahi. ✅

Worked example Example 8 — Case I: exam twist (lazy variable capture)

funcs = [lambda: i for i in range(3)]        # functions ki list
print([f() for f in funcs])

lambda: i ek chhota anonymous function hai jo i return karta hai. Yeh woh twist hai jo exam writers love karte hain. Yeh khud generator expression nahi hai, lekin yeh wohi "baad mein evaluate, abhi nahi" principle expose karta hai jo laziness ke core mein hai — isliye yeh matrix ke tricky corner mein belong karta hai.

Forecast: tum chahoge answer karo [0, 1, 2]. Ruko. Kya actually print hota hai?

i kab read hota hai? Har lambda i ki value creation par capture nahi karta; woh variable i ko capture karta hai, aur use sirf tab read karta hai jab call ho. Yeh step kyun? Yeh "call time par value read karna" exactly wahi deferred-computation idea hai jo generators embody karte hain. Trap yeh assume karna hai ki value tab freeze ho jaati hai jab lambda banta hai.
Loop khatam hone ke baad call karo. Jab tak f() run hota hai, loop khatam ho chuki hoti hai aur i apni final value 2 rakhti hai teeno lambdas ke liye. Yeh step kyun? Teeno ek variable i share karte hain; loop use 2 par chhod jaata hai. Deferred read → sab 2 dekhte hain.

Verify: [2, 2, 2] print hota hai, not [0, 1, 2]. [0,1,2] paane ke liye fix: default arg se eagerly bind karo, lambda i=i: i. ✅

Active Recall

Recall Answers cover karo aur self-test karo

sum(x*x for x in range(5)) se kya print hota hai? ::: 30 list(n**3 for n in range(6) if n % 2 == 0) kya deta hai? ::: [0, 8, 64] list(g) se g fully consume hone ke baad, sum(g) kya return karta hai? ::: 0 — g exhausted ho gaya (m*m for m in count(1)) par islice kyun use karna padta hai? ::: Source infinite hai; plain list kabhi finish nahi hoti aur memory khatam ho jaati funcs = [lambda: i for i in range(3)] mein, [f() for f in funcs] kya print karta hai? ::: [2, 2, 2] — i call time par read hota hai, jab loop pehle hi khatam ho chuki hoti hai any("FATAL" in line for line in f) file padhna kab band karta hai? ::: Pehli matching line par — yeh short-circuit karta hai

Connections

Generator expressions — memory efficiency — parent idea jise yeh page drill karta hai
Lazy evaluation — "computed on demand" Examples 4, 6, 8 ko power karta hai
Iterators and the iterator protocol — next() aur forward-only pointer
Generator functions and yield — def/yield cousin, same laziness
List comprehensions — eager alternative jo Examples 2, 8 mein use hua
Big-O space complexity — $O (1)$ vs $O (n)$ jo pehle figure mein dikhaya gaya
Memory management in Python — kyun huge-file case (Ex 7) matter karta hai