6.1.5 · D5 · HinglishParallelism & Multicore
Question bank — Shared memory vs distributed memory
6.1.5 · D5· Hardware › Parallelism & Multicore › Shared memory vs distributed memory
Recall Pehle, ek-sentence anchors (sirf tab dekho jab neeche ke words unfamiliar lagein)
Shared memory = har processor simple load/store se har byte ko naam de sakta hai aur use access kar sakta hai. Distributed memory = ek processor sirf apne bytes ko touch kar sakta hai; doosre ka data paane ke liye use ek message bhejna padta hai. Cache coherence = hardware ka yeh wada ki agar main ek value likhta hoon, toh koi doosra cache purani value nahi dikhata rahega. Latency = koi bhi data aane se pehle ki fixed wait; bandwidth = jab pipe khul jaati hai tab bytes kitni tezi se stream hoti hain.
True or false — justify karo
Shared memory ka matlab hai ki machine mein literally sirf ek physical RAM bank hai.
False — iska matlab ek logical address space hai; NUMA machines mein kai physical memory banks hote hain, lekin har CPU phir bhi unhe address kar sakta hai (kuch bas door/slow hote hain). Dekho NUMA Architecture.
Distributed-memory system mein, ek processor doosre processor ki variable ko read kar sakta hai agar use address pata ho.
False — addresses local hain; same numeric address har node par alag memory ko refer karta hai. Andar jaane ka ek hi tarika hai — Message Passing Interface (MPI) se explicit message.
Cache coherence shared-memory code mein locks ki zaroorat khatam kar deta hai.
False — coherence sirf individual cache lines ko consistent rakhta hai, lekin
x = x+1 jaisa read-modify-write kai operations ka hota hai; do threads phir bhi interleave ho sakte hain aur ek update kho sakti hai. Iske liye tumhe atomics/locks chahiye.Message passing hamesha shared-memory load se slower hoti hai.
Ek operation ke liye True (µs vs ns), lekin ek system claim ke roop mein false — shared memory kuch dozen cores ke baad bus/coherence bottleneck mein phans jaata hai, jabki message passing hazaaron nodes tak scale hoti rehti hai.
False sharing tab hoti hai jab do threads same variable likhte hain.
False — yeh tab hoti hai jab woh alag variables likhte hain jo same 64-byte cache line mein aate hain; hardware poori line ko invalidate karta hai, byte ko nahi. Dekho False Sharing.
OpenMP programs bina kisi badlaav ke distributed-memory clusters par run hote hain.
False — OpenMP ek single shared address space (threads jo memory share karte hain) assume karta hai; alag nodes par tumhe message passing chahiye. Hybrid MPI+OpenMP aam fix hai.
Distributed system mein bilkul bhi cache coherence problem nahi hoti.
Hardware coherence ke liye True — har node ki memory private hai, toh cross-node invalidations nahi hote. Iska equivalent burden programmer ke upar aa jaata hai ki woh messages ke zariye replicated copies ko sync mein rakhe.
Shared-memory machine mein cores badhane se throughput hamesha badhti hai.
False — jis point par memory bus ya coherence traffic saturate ho jaata hai uske baad, cores add karne se contention badhti hai, aur throughput flat ho sakti hai ya gir bhi sakti hai.
Error dhundho
"Matrix B ko saare nodes par broadcast karne ka cost ek message hai, toh yeh sasta hai."
N-1 doosre nodes tak pahunchne ke liye tum N-1 messages bhejte ho (ya unka ek log-depth tree); har ek latency pay karta hai. Yeh ek baar pay hota hai, lekin yeh single message nahi hai."Kyunki har thread alag i ke liye count[i] likhta hai, toh koi coherence traffic nahi hai."
Agar
count[] entries ek cache line share karti hain, toh har write doosron ki copies ko invalidate kar deta hai — classic false sharing. Alag index ≠ alag cache line."100 GPUs tak scale karne ke liye hum NVLink par ek bada shared-memory space use karte hain."
Koi single machine itne GPUs ko coherently host nahi karti; itne saare mein coherence maintain karna impractical hai. Tum distributed memory + AllReduce over the interconnect use karte ho.
"1 KB ke 1000 messages bhejne ka cost 1 MB ke ek message ke barabar hai kyunki bytes equal hain."
Bytes equal hain lekin latency per-message hai: 1000 × latency vs 1 × latency. Chhote messages mein latency dominate karti hai, toh ek bade message mein batching bahut tez hoti hai.
"MSI mein, Shared line par write free hai kyunki copies pehle se exist karti hain."
Iska ulta — Shared line par write ke liye har doosre holder ko invalidate karna padta hai (up to
N-1 bus transactions). Shared hona write ko expensive banata hai."Blocking receive() time waste karta hai, toh non-blocking hamesha better hai."
Non-blocking tumhe compute aur communication overlap karne deta hai, lekin iske liye baad mein
wait/test aur sahi buffer management chahiye. Laparwahi se use karne par yeh half-arrived data read kar leta hai — "better" tabhi hai jab tum actually useful work overlap karo.Why questions
Coherence traffic contended write par roughly kyun scale karta hai?
Ek writer ko baaki cache holders ko notify karna padta hai ki woh apni copies invalidate kar dein, toh message count processors ki sankhya ke saath badhta hai jo us line ko share karte hain.
MPI programmers jaanbujhkar communication ko batch kyun karte hain?
Kyunki mein ek fixed per-message latency term hai; kam, bade messages us fixed cost ko amortize karte hain iske bajaye ki hazar baar pay karo.
Spatial locality (nearby addresses) ek single thread ke liye achhi cheez kyun hai lekin shared memory mein ek khatara hai?
Ek thread ke liye, data ko ek line mein pack karna kam misses deta hai. Kai threads ke liye jo nearby addresses likhte hain, wahi line ek coherence ping-pong ban jaati hai — jo closeness ek thread ki madad karti thi woh kai threads ko hurt karti hai. Dekho False Sharing.
Distributed memory tab bhi scale karta rehta hai jab shared memory ruk jaata hai, kyun?
Distributed nodes ki independent memories hain aur koi shared bus nahi hai jo serialize kare; nodes add karne se bandwidth add hoti hai. Shared memory sabko ek bus/coherence fabric se funnel karta hai jo saturate ho jaata hai.
Weaker consistency model aksar shared-memory hardware ko faster kyun banata hai?
Ek weaker model hardware ko writes ko reorder/buffer karne deta hai iske bajaye ki har write turant globally visible ho, stalls cut hoti hain — is keemat par ki programmer explicit fences daale jahan ordering truly matter karta hai.
Distributed memory programming shared memory se zyada mushkil kyun maani jaati hai?
Tumhe haath se orchestrate karna padta hai ki kaun kya bhejta hai, kab, aur kisko, aur mismatched send/receive pairs deadlock cause karte hain — woh kaam jo shared-memory hardware implicitly loads/stores ke saath karta hai.
Edge cases
Shared-memory write par kya hota hai jab line pehle se tumhare apne cache mein Modified state mein hai?
Koi invalidation nahi chahiye — tum pehle se ek-maatra up-to-date copy hold karte ho, toh yeh ~1 transaction cost karta hai, nahi.
Distributed memory mein communication ka cost kya hai agar do processes koi data share nahi karti?
Zero communication cost — ek embarrassingly parallel workload sirf local compute pay karta hai. Yeh woh best case hai jo distributed memory target karta hai.
NUMA machine par, kya har "shared" access equally fast hoti hai?
Nahi — apna local memory bank access karna fast hai; remote bank tak pahunchna interconnect cross karta hai aur slower hai. Yeh abhi bhi ek address space hai, lekin non-uniform latency ke saath. Dekho NUMA Architecture.
Zero-byte MPI message ki latency kya hai?
Zero nahi — yeh essentially hai, kyunki term vanish ho jaata hai lekin fixed setup/OS/NIC overhead rehta hai. Isi tarah latency aksar measure ki jaati hai.
Agar distributed AllReduce saare nodes par gradients sum karta hai, toh kya hota hai jab sirf ek node slow ho (straggler)?
Poora collective ruk jaata hai — har node synchronization point par wait karta hai jab tak slowest nahi aa jaata, toh ek straggler sab ko throttle karta hai. Isliye load balance utna hi important hai jitna raw speed.
Ek single-core machine kya consistency free mein deti hai jo ek multicore shared-memory machine nahi deti?
Apne khud ke program order ki sequential consistency ek core par automatic hai; kai cores ke saath, alag cores ke writes alag orders mein dikh sakte hain jab tak model aur fences use enforce nahi karte.