Batch vs real-time inference

WHAT are we even choosing between?

WHY does the distinction exist? (first principles)

Every prediction has two clocks:

1. When the input becomes available (arrival time $t_a$).
2. When the prediction is actually needed (deadline $t_d$).

Define the slack: $\text{slack} = t_d - t_a$

• If slack is large (you know the input long before you need the answer) → precompute in batch. Nothing is wasted by doing it early... unless the answer goes stale.
• If slack is tiny (input arrives the instant you need the answer) → you must serve real-time; there's no earlier moment to precompute.

HOW to reason about cost & throughput (derive it)

Let:

• $N$ = number of predictions needed per day,
• $c$ = compute cost per prediction (forward pass),
• $u$ = fraction of predictions actually used by a consumer.

Batch cost per useful prediction, if you predict for all $N$ candidates but only $uN$ get used: $C_{\text{batch}} = \frac{N \cdot c}{u \cdot N} = \frac{c}{u}$

Why this step? You pay $Nc$ to compute everything, but only $uN$ predictions do work, so cost is spread over the useful ones. If $u$ is small (you precompute for users who never log in), batch is wasteful.

Real-time cost per useful prediction — you only compute when asked, so $u \approx 1$: $C_{\text{rt}} = c + o$ where $o$ is the per-request overhead (network, serialization, keeping a warm server idle).

Break-even: batch wins when $\frac{c}{u} < c + o \iff \frac{1}{u} < 1 + \frac{o}{c} \iff u > \frac{c}{c+o}$

Why this matters: low overhead $o$ (small $\frac{o}{c}$) pushes the threshold near 1 → real-time is cheap → lean real-time. High utilization near 1 with expensive models → batch precompute and cache.

Throughput vs Latency (they trade off)

Batch systems group inputs to use vectorized/GPU parallelism, maximizing throughput but adding wait-to-fill-a-batch latency. Real-time can use dynamic batching (wait a few ms to collect concurrent requests) as a compromise — the middle ground.

Staleness — the hidden cost of batch

Flashcards

Connections

• Model Serving — the infra that hosts real-time endpoints.
• Feature Store — serves precomputed batch features to real-time models (bridges both worlds).
• Latency and Throughput — the trade-off this note formalizes.
• Autoscaling — how real-time absorbs traffic spikes.
• Data Drift and Staleness — why batch predictions decay.
• Dynamic Batching — the compromise between the two extremes.
• Caching Strategies — batch results are essentially a warm cache.

Concept Map

Hinglish (regional understanding)