Batch vs real-time inference: Visual walkthrough

We will use a few symbols. Before any of them appear in an equation, we name them here in plain words:

Definition The five quantities (memorize these — everything is built from them)

$N$ ::: how many predictions you could need in a day (the whole candidate pool).
$c$ ::: the cost of one forward pass through the model — one prediction. Think "one coin." We always assume $c > 0$ (a real prediction costs something).
$u$ ::: utilization — the fraction of your precomputed (batch) predictions that someone actually looks at. A number strictly between 0 and 1. If $u = 0.6$ , then 60% get used, 40% are wasted.
$o$ ::: the overhead of answering live — the network hop, packing the request, keeping a warm server idle waiting for callers. Paid once per real-time answer.
slack $= t_{d} - t_{a}$ ::: how long before you need the answer ( $t_{d}$ , deadline) you already have the input ( $t_{a}$ , arrival). Big slack = you knew early.

Everything below is a story about these five.

Step 1 — The two clocks that force the whole choice

WHAT. Every prediction lives between two moments: the instant its input becomes known ( $t_{a}$ , "arrival"), and the instant its answer is due ( $t_{d}$ , "deadline"). The gap between them is the slack.

WHY this and not cost first? Because slack can settle the question before any arithmetic. If the answer is needed the exact instant the input appears, there is no earlier moment in which to precompute — batch is physically impossible. Cost only matters when you actually have a choice.

PICTURE. Two timelines. On top, a fraud swipe: arrival and deadline sit on top of each other — zero slack, one option. On the bottom, a recommendation: the input is known hours before the user opens the app — a wide green slack bar you could fill with precomputation.

Figure 1 — The slack gate: zero slack forces real-time; wide slack (green bar) makes batch possible.

$slack = when answer is due t_{d} - when input is known t_{a}$

$t_{d} - t_{a}$ small (near zero) → real-time is forced, no further analysis.
$t_{d} - t_{a}$ large → you may batch. Now, and only now, cost decides. Proceed to Step 2.

Step 2 — The naive cost of one prediction

WHAT. Both strategies run the identical model, so one forward pass costs the same $c$ in both. Draw $c$ as a single coin-height bar. This is our unit of everything.

WHY. We must anchor the comparison to something both sides share, so the difference between them is visible cleanly. That shared thing is $c$ , the cost of a forward pass. Everything else is a multiplier or an add-on to this one coin.

PICTURE. A single blue bar of height $c$ , labeled "one forward pass — same weights, same math, both modes." This is the mistake the parent warned about ("real-time = batch size 1") shown honestly: the math really is one bar. The difference lives in the system around it, which the next steps draw.

Figure 2 — The shared unit: one forward pass costs $c$ (assume $c > 0$ ) in both modes.

Step 3 — Batch's hidden waste: divide by utilization

WHAT. In batch you compute all $N$ candidates, paying $N \cdot c$ . But only a fraction $u$ of them get consumed — that's $u N$ useful predictions. The honest cost per useful answer spreads the total bill over only the useful ones.

WHY divide, not subtract? Because we want a fair per-answer price to compare against real-time's per-answer price. You paid for $N$ ; you got value from $u N$ . The price of each valuable answer is total-cost ÷ valuable-count — that is division.

$C_{batch} = \frac{N \cdot c pay for ALL candidates}{only these are used u \cdot N} = \frac{c}{u}$

$N$ cancels — the size of the pool doesn't matter, only the fraction wasted does.
This division is only defined because $u > 0$ (see the definitions box). $u = 0$ — nothing precomputed is ever used — is a forbidden division-by-zero: the "cost per useful answer" is genuinely undefined because there are no useful answers to spread the bill over.
As $u \to 1$ (everyone's prediction gets used) → $C_{batch} \to c$ : perfect, no waste.
As $u \to 0^{+}$ (almost nothing gets used) → $C_{batch} \to \infty$ : you precomputed a mountain nobody touched.

PICTURE. A pool of $N$ circles; only a $u$ -fraction glow (used), the rest are dim (wasted). Beside it, the bar $c$ stretched taller by the factor $\frac{1}{u}$ — the shorter the glowing slice, the taller the true per-answer cost climbs.

Figure 3 — Batch divides its bill by utilization $u$ ; low $u$ stretches the per-answer cost $c / u$ sky-high.

Step 4 — Real-time's hidden tax: the overhead $o$

WHAT. Real-time only computes when asked, so nothing is wasted. Its own utilization — call it $u_{rt}$ — is exactly 1: every answer it produces was explicitly requested, so none is discarded. But every live answer carries an extra cost $o$ : the network round-trip, packing/unpacking the request, and keeping a warm server idling so there's no cold start.

WHY add, not multiply? Because overhead is a fixed toll paid once per request, on top of the forward pass — it stacks on, it doesn't scale the compute. Addition = "and also this."

$C_{rt} = the forward pass c + network + serialize + warm idle o$

No "divide by utilization" appears — because $u_{rt} = 1$ exactly: dividing by 1 changes nothing, so the term vanishes. (Contrast batch, where $u < 1$ inflated the cost.)
If $o \to 0$ (overhead negligible) → $C_{rt} \to c$ : real-time is as cheap as a lone forward pass.
$o$ is what you pay for freshness on demand.

PICTURE. The blue $c$ bar with a red $o$ stacked on top of it — a two-segment tower labeled "compute + toll." No waste, but a permanent surcharge.

Figure 4 — Real-time wastes nothing ( $u_{rt} = 1$ ) but stacks a fixed overhead $o$ on top of $c$ .

Step 5 — Put the two towers side by side and let them cross

WHAT. We now have two per-answer prices: $C_{batch} = \frac{c}{u}$ and $C_{rt} = c + o$ . Batch wins on cost exactly when its tower is shorter:

$\frac{c}{u} < c + o$

WHY an inequality? We don't want a single number — we want the boundary where the cheaper strategy flips. An inequality carves the world into "batch-land" and "real-time-land." Solving for $u$ tells us how much utilization you need to justify precomputing.

Solve it, one honest algebra step at a time. Recall from the definitions that $c > 0$ and $u > 0$ — both facts are needed to keep the inequality's direction. Divide both sides by $c$ (positive, so the direction is safe):

$\frac{1}{u} < 1 + \frac{o}{c} ⟹ u > \frac{1}{1 + \frac{o}{c}} = \frac{c}{c + o}$

Left move: divided by $c$ → turns absolute costs into ratios (the $\frac{o}{c}$ is "overhead measured in forward-passes"). Legal because $c > 0$ .
Right move: flipped the reciprocal → an inequality $\frac{1}{u} < X$ becomes $u > \frac{1}{X}$ . This flip is only valid because $u > 0$ (both sides positive); dividing/reciprocating across a positive quantity preserves the direction, whereas a zero or negative $u$ would break it.

PICTURE. Two curves against $u$ on the horizontal axis: a flat red line at height $c + o$ (real-time, independent of $u$ ) and a falling blue curve $\frac{c}{u}$ (batch, cheaper as $u$ grows). They cross at $u^{*} = \frac{c}{c + o}$ . Left of the crossing (low utilization) the blue curve is above the red line → real-time wins. Right of it → batch wins. The crossing point is shaded.

Figure 5 — The two cost curves cross at $u^{*} = \frac{c}{c + o}$ : right of it batch wins, left of it real-time wins.

Step 6 — Read the threshold's two extremes (degenerate cases)

WHAT. A formula is only trusted once you push it to its limits and it still behaves. We test $\frac{c}{c + o}$ at $o \to 0$ and $o \to \infty$ .

WHY. Edge cases are where wrong formulas break. If ours gives sane answers at the extremes, the whole derivation is validated.

$lim_{o \to 0} \frac{c}{c + o} = \frac{c}{c} = 1, lim_{o \to \infty} \frac{c}{c + o} = 0$

$o \to 0$ (real-time is almost free): threshold $\to 1$ . You'd need near-perfect utilization to justify batch — so lean real-time. Matches intuition: if being live costs nothing, always be live.
$o \to \infty$ (real-time infra is ruinously expensive): threshold $\to 0$ . Almost any utilization justifies batch — so precompute and cache aggressively.
The special case $u = 1$ (everything precomputed is used) is always $\geq$ any threshold below 1, so batch wins whenever there's slack. This is the recommendation case.

PICTURE. A slider of $o$ from 0 to large; the threshold marker slides from 1 (all the way right, "real-time land dominates") down to 0 (all the way left, "batch land dominates"). Two annotated snapshots.

Figure 6 — The batch threshold $c / (c + o)$ slides from 1 (as $o \to 0$ ) down to 0 (as $o \to \infty$ ).

Step 7 — The hybrid escape hatch (why the real world isn't binary)

WHAT. Real pipelines split themselves: the slow-changing, expensive part is batched; the query-specific part is served live. A search engine batches document embeddings nightly, then does a real-time nearest-neighbor lookup at query time.

WHY. Because different parts of one pipeline have different slack. Embeddings of documents don't depend on your query → huge slack → batch. Matching your query to them exists only when you type → zero slack → real-time. Apply Steps 1–6 per component, not to the whole.

PICTURE. A pipeline splitting into two lanes: a blue "batch lane" (high slack, high utilization, Feature Store / Caching Strategies) feeding a red "real-time lane" (zero slack) at query time, using Dynamic Batching to claw back throughput.

Figure 7 — Hybrid pipeline: batch the slow, stable part; serve the live, query-specific part in real-time.

The one-picture summary

WHAT. One figure compresses the whole walkthrough: the slack gate first (does a choice even exist?), then the two cost towers and their crossing at $u^{*} = \frac{c}{c + o}$ , then the drift/staleness veto.

Figure 8 — The whole derivation in one frame: slack gate → cost crossing at $u^{*} = c / (c + o)$ → staleness veto.

Recall Feynman retelling — the whole walkthrough in plain words

Start with two alarm clocks. One rings when your ingredient arrives, the other when the meal is due. If they ring at the same time, you have no choice — cook on the spot (real-time). If the first rings hours early, you could cook ahead (batch), so now money decides. Cooking ahead means making a whole tray of $N$ dishes even though only a fraction $u$ get eaten — so each eaten dish secretly cost you $\frac{c}{u}$ , more than the plain $c$ because of the wasted trays. Cooking on the spot wastes nothing, but you pay a doorman-and-kitchen tax $o$ for keeping the stove hot all day, so each dish costs $c + o$ . Draw both prices as $u$ (how much gets eaten) grows: the on-the-spot price is a flat line at $c + o$ ; the cook-ahead price is a curve that starts sky-high (nobody eats → infinite waste) and falls as more gets eaten. They cross at exactly $u = \frac{c}{c + o}$ . To the right — lots gets eaten — cook ahead. To the left — most is wasted — cook on the spot. Push the tax $o$ to zero and the crossing slides to 1 (only bother cooking ahead if everyone eats). Push $o$ huge and it slides to 0 (cook ahead almost always). And one last warning your formula can't see: a dish cooked at midnight and served at 6 PM might be soggy — staleness can veto even the cheapest plan.

Connections

Batch vs real-time inference — the parent topic this walkthrough visualizes.
Latency and Throughput — the trade-off hiding behind $o$ and batch's fill-wait.
Feature Store — how the hybrid batch lane feeds the real-time lane.
Dynamic Batching — how real-time claws back throughput inside the red lane.
Caching Strategies — where precomputed batch answers live for cheap lookup.
Data Drift and Staleness — the veto that overrides a winning cost argument.
Autoscaling & Model Serving — the infra whose idle cost is the overhead $o$ .