Batch vs real-time inference — Hinglish notes

5.3.7 · AI-ML › MLOps & Deployment

WHAT — hum choose kya kar rahe hain?

WHY — yeh distinction exist kyun karta hai? (first principles)

Har prediction ke do clocks hote hain:

Jab input available hoti hai (arrival time $t_{a}$ ).
Jab prediction actually chahiye hoti hai (deadline $t_{d}$ ).

Slack define karo: $slack = t_{d} - t_{a}$

Agar slack bada hai (tumhe input answer ki zaroorat se pehle pata hai) → batch mein precompute karo. Pehle se karna wasteful nahi hai... jab tak answer stale na ho jaaye.
Agar slack tiny hai (input exactly usi waqt aata hai jab answer chahiye) → real-time serve karna zaroori hai; precompute karne ka koi pehle ka moment hai hi nahi.

HOW — cost & throughput ke baare mein reason karna (derive karo)

Maano:

$N$ = din bhar mein chahiye predictions ki sankhya,
$c$ = compute cost per prediction (forward pass),
$u$ = fraction of predictions jo actually consumer use karta hai.

Batch cost per useful prediction, agar tum saare $N$ candidates ke liye predict karo lekin sirf $u N$ use ho: $C_{batch} = \frac{N \cdot c}{u \cdot N} = \frac{c}{u}$

Yeh step kyun? Tum $N c$ pay karte ho sab compute karne ke liye, lekin sirf $u N$ predictions kaam karti hain, toh cost useful waale pe spread hoti hai. Agar $u$ chhota hai (tum unhe precompute karte ho jo kabhi log in hi nahi karte), toh batch wasteful hai.

Real-time cost per useful prediction — tum sirf tab compute karte ho jab maanga jaaye, toh $u \approx 1$ : $C_{rt} = c + o$ jahan $o$ hai per-request overhead (network, serialization, warm server idle rakhna).

Break-even: batch tab jeetta hai jab $\frac{c}{u} < c + o ⟺ \frac{1}{u} < 1 + \frac{o}{c} ⟺ u > \frac{c}{c + o}$

Yeh kyun matter karta hai: kam overhead $o$ (chhota $\frac{o}{c}$ ) threshold ko 1 ke paas push karta hai → real-time sasta hai → lean real-time. High utilization near 1 mahenge models ke saath → batch precompute karo aur cache karo.

Throughput vs Latency (yeh trade off karte hain)

Batch systems inputs group karte hain vectorized/GPU parallelism use karne ke liye, throughput maximize karte hain lekin batch fill hone ka wait latency add hoti hai. Real-time dynamic batching use kar sakta hai (concurrent requests collect karne ke liye kuch ms wait karo) as a compromise — yeh beech ka raasta hai.

Staleness — batch ka hidden cost

Flashcards

Batch vs real-time decide karne ke liye sabse best single quantity kya hai?

Slack

t_{d} - t_{a}

: deadline se pehle input kitna pata hai. Zero slack real-time force karta hai.

Batch inference define karo.

Ek schedule pe accumulated bade set pe predict karna, results baad mein lookup ke liye store karna (offline).

Real-time inference define karo.

Request arrival pe tight latency budget ke andar synchronously prediction compute karna (online/on-demand).

"Real-time = batch with batch size 1" galat kyun hai?

Model math identical hai lekin real-time ko live serving infra chahiye: autoscaling, load balancing, p99 SLAs, cold-start handling.

Agar sirf fraction

u

use ho toh batch cost per useful prediction?

c / u

— total compute

N c

u N

actually consumed predictions pe divide karo.

Cost mein batch real-time ko kab beat karta hai?

Jab utilization

u > \frac{c}{c + o}

ho, yaani utilization compute-to-compute-plus-overhead ratio se zyada ho.

Latency vs throughput?

Latency = ek request ka time; throughput = predictions per second. Real-time latency optimize karta hai, batch throughput optimize karta hai.

Dynamic batching kya hai?

Real-time servers thodi der (kuch ms) wait karte hain concurrent requests group karne ke liye, thodi latency ke badle mein kaafi zyada throughput milta hai.

Batch ki fundamental weakness?

Staleness — pehle compute ki gayi predictions tab tak outdated ho sakti hain jab tak use hon.

Sab kuch precompute kyun nahi kar sakte?

Input (feature combination) live event hone tak exist hi nahi karta (jaise yeh exact transaction), aur input space effectively infinite ho sakta hai.

Recall Feynman: 12-saal ke bachche ko explain karo

Sandwich banana socho. Batch = tum subah 100 sandwiches banate ho aur fridge mein rakh dete ho, toh lunch fast hota hai — lekin sandwich soggy (stale!) ho sakti hai aur kuch kabhi khayi hi nahi jaatein (waste!). Real-time = jab bhi koi order kare tab sandwich banate ho — hamesha fresh, kuch waste nahi, lekin insaan ko wait karna padta hai aur tumhe ek cook poora din ready rakhna padta hai. Agar pata ho sab kya chahte hain pehle se aur sab khayenge → pehle bana lo (batch). Agar har order ek surprise ho jo andaaza nahi laga sakte → turat banao (real-time).

Connections

Model Serving — real-time endpoints host karne ki infra.
Feature Store — real-time models ko precomputed batch features serve karta hai (dono worlds ko bridge karta hai).
Latency and Throughput — yeh note jis trade-off ko formalize karta hai.
Autoscaling — real-time traffic spikes kaise absorb karta hai.
Data Drift and Staleness — batch predictions kyun decay hoti hain.
Dynamic Batching — dono extremes ke beech ka compromise.
Caching Strategies — batch results essentially ek warm cache hain.