Batch vs real-time inference: Worked examples

Before anything, let me re-anchor every symbol so you never have to scroll up:

Definition Two more symbols this page needs: the batch window and staleness tolerance

$W$ = batch window = the block of wall-clock time you reserve to run the whole batch job (e.g. the "overnight" span from midnight to 6 AM $\Rightarrow W = 6 h = 21, 600$ s). It must sit inside the slack: you can only precompute in the gap between when the inputs exist and when the answers are needed, so $W \leq slack$ . Across $M$ parallel machines the effective compute time available is $M \cdot W$ , so the feasibility test is $N \cdot (time per prediction) \leq M \cdot W$ .
$S$ = staleness tolerance = the maximum age (in hours) a precomputed answer may reach before it is considered wrong enough to matter. If the gap between compute-time and use-time, $Δ t$ , satisfies $Δ t \leq S$ , staleness is acceptable; if $Δ t > S$ , batch's answer is stale and Gate 2 should reject batch. "Staleness OK" in Gate 2 means exactly $Δ t \leq S$ .

The scenario matrix

Every situation this topic throws at you falls into one of these cells. The examples below are labelled with the cell they cover.

Cell	Axis / edge case	Governing quantity
A	Huge slack, high $u$	slack large and $u > \frac{c}{c + o}$ → batch
B	Input born at the deadline (no earlier tick)	precompute infeasible → real-time forced
C	Large slack but low $u$	cost rule $u > \frac{c}{c + o}$ fails
D	Break-even boundary ( $u = \frac{c}{c + o}$ )	tie-breaker (staleness/latency)
E	Degenerate: $o = 0$ (overhead-free)	threshold $\to 1$
F	Degenerate: $o \to \infty$ (huge warm-up)	threshold $\to 0$
G	Limiting: infinite input space	can't precompute at all
H	Hybrid split by "what changes slowly"	pipeline decomposition
I	Throughput vs latency numeric trade	Dynamic Batching + queue stability
J	Exam twist: staleness beats cost	slack large but answer decays ( $Δ t > S$ )

We now walk one example per cell. Every number is machine-checked at the bottom.

The decision flow (walk it in words, then the figure)

Here is the logic in plain words, so you can follow it even without the picture:

Start the moment the input arrives at $t_{a}$ .
Gate 1 — feasibility: Does the input exist early enough to work ahead — i.e. is there a batch window $W > 0$ before the deadline in which the input already exists? If no (the input only appears at the deadline itself), take the red forced path straight to real-time — no cost math is possible, because you cannot compute something that hasn't happened yet.
If yes, drop to Gate 2 — economics: Is $u > \frac{c}{c + o}$ and is staleness tolerable ( $Δ t \leq S$ )? If yes → batch. If no → real-time.

Figure s01 — Decision flow (described in full so it stands alone). A top black box "input arrives ( $t_{a}$ )" feeds down into Gate 1 ("input exists early?"). From Gate 1 a red arrow labelled NO drops to a red box "REAL-TIME (forced)" — this is the fast exit when precompute is infeasible. A "YES" arrow from Gate 1 goes right into Gate 2 (" $u > c / (c + o)$ ?"), which splits into "no → REAL-TIME" and "yes → BATCH". Trace the red arrow first: it is the only path that skips all cost arithmetic.

The examples

Worked example Example 1 — Cell A: huge slack, high utilization (with the cost check)

Setup: A streaming service scores $N = 200, 000, 000$ users nightly for "top picks". Recs are needed only when a user opens the app (any time next day), and $60%$ of users open it. Model = $5$ ms/user, with per-prediction compute $c = \$ 1\times10^{-4} $an d r e a l - t im eo v er h e a d$ o = $3\times10^{-4} $(s am e w a r m - ser v er in f r aa s E x 3) . O v er ni g h t ba t c h w in d o w$ W = 6\text{ h} = 21{,}600$ s.

Forecast: Guess now — batch or real-time? (Write it down.)

Compute the slack. Input (yesterday's watch history) exists at midnight; deadline is whenever the user opens the app — hours later. So slack $\approx$ several hours $≫ 0$ . Gate 1 passed (input exists early → precompute feasible). Why this step? Gate 1 is first: if precompute were infeasible we'd stop and pick real-time.
Compute utilization. $u = 0.60$ . Why this step? We now need the economics gate, which requires $u$ .
Cost threshold. $\frac{c}{c + o} = \frac{1 \times 1 0 ^{- 4}}{1 \times 1 0 ^{- 4} + 3 \times 1 0 ^{- 4}} = \frac{1}{4} = 0.25$ . Check: $u = 0.60 > 0.25$ → batch cheaper. Why this step? This is the concrete cost check — "high $u$ " only means "batch" because $0.60$ clears the $0.25$ bar. Never assert without comparing.
Batch-window feasibility. Total compute $= N \times 5 ms = 2 \times 1 0^{8} \times 5 \times 1 0^{- 3} s = 1 0^{6}$ s of GPU-time. Using the feasibility test $N \cdot (time) \leq M \cdot W$ : we need $M \geq \frac{1 0 ^{6}}{21 , 600} \approx 46.3$ , so $M = 47$ machines fit the job inside the $6$ -hour window. Why this step? Confirms the workload physically fits the batch window $W$ once you fan it out across $M$ machines — a "batch wins on cost" answer is useless if the job can't finish before it's needed.

Answer: Batch. Gate 1 passed, $u = 0.60 > 0.25$ , staleness of hours OK ( $Δ t \leq S$ ) → Cell A.

Verify: $\frac{c}{c + o} = 0.25$ and $0.60 > 0.25$ ; $C_{batch} = c / u = 1 0^{- 4} /0.6 \approx 1.67 \times 1 0^{- 4} < C_{rt} = 4 \times 1 0^{- 4}$ ; total compute $= 2 \times 1 0^{8} \times 5 \times 1 0^{- 3} = 1 0^{6}$ s; machines $⌈ 1 0^{6} /21600 ⌉ = 47$ . ✓

Worked example Example 2 — Cell B: input born

at the deadline (precompute infeasible) Setup: A card swipe must be approved/denied in $< 50$ ms. The fraud features (this device, this location, this exact amount) only exist at swipe time.

Forecast: Can you precompute this? Yes / No?

Separate two different things. The $50$ ms is the response budget $t_{d} - t_{a} = 50 ms$ : the tiny time you have to answer after the input arrives. Cell B is not about that budget being zero — it is about whether any batch window $W > 0$ existed before the input, in which the input already existed. Those are different clocks. Why this step? The reviewer's trap: a positive response budget does not imply you can precompute. Precompute feasibility asks whether the input existed before the moment you'd start batching.
Apply feasibility. The features are born at the swipe, i.e. at $t_{a}$ = the event itself. There is no earlier tick where they existed, so no batch window $W > 0$ can contain them and Gate 1 fails — regardless of the $50$ ms response budget. This is the true "input = arrival = start of the only possible answer" case that names Cell B. Why this step? You cannot compute a prediction whose input hasn't been created yet — this is the red forced exit.
Cost rule? Not evaluated — infeasibility short-circuits Gate 2 entirely.

Answer: Real-time, forced. Cell B.

Verify: batch window available before input, $W = 0$ (input born at arrival) $\Rightarrow$ feasibility False; the response budget $50$ ms is a separate positive number. Both facts hold independently. ✓

Worked example Example 3 — Cell C: large slack but low utilization

Setup: You precompute product recs for all $N = 10, 000, 000$ registered users nightly, but only $2%$ log in on a given day. Model cost $c = \$ 1\times10^{-4} $p er p r e d i c t i o n . R e a l - t im eo v er h e a d$ o = $3\times10^{-4}$ per request.

Forecast: Slack is big — but does batch still win?

Utilization $u = 0.02$ . Why this step? Low $u$ is exactly the danger case: batch pays for the $98%$ that's never seen.
Threshold $\frac{c}{c + o} = \frac{1 \times 1 0 ^{- 4}}{1 \times 1 0 ^{- 4} + 3 \times 1 0 ^{- 4}} = \frac{1}{4} = 0.25$ . Why this step? The rule says batch wins iff $u > \frac{c}{c + o}$ . Compute the bar first.
Compare (strict " $>$ "). $u = 0.02$ , and $0.02 > 0.25$ is false. So batch loses despite huge slack. Why this step? Slack tells you batch is possible; the strict cost inequality tells you whether it's worth it.

Answer: Real-time. Big slack, but wasted compute makes on-demand cheaper. Cell C.

Verify: $C_{\text{batch}} = c/u = 10^{-4}/0.02 = \$ 5\times10^{-3} $p er u se f u l p r e d i c t i o n;$ C_{\text{rt}} = c + o = $4\times10^{-4} $. S in ce$ 5\times10^{-3} > 4\times10^{-4} $, r e a l - t im e i sc h e a p er, ma t c hin g$ u<0.25$. ✓

Worked example Example 4 — Cell D: exactly at break-even

Setup: Same $c = 1 \times 1 0^{- 4}$ , $o = 3 \times 1 0^{- 4}$ , but now utilization is exactly the threshold: $u = 0.25$ .

Forecast: A tie — what decides it?

Threshold $= \frac{c}{c + o} = 0.25$ (from Example 3).
Apply the strict rule. The decision rule "batch wins iff $u > \frac{c}{c + o}$ " uses a strict " $>$ ". Here $u = 0.25 = \frac{c}{c + o}$ , so " $u > 0.25$ " is false — batch does not win. But plugging in shows the two costs are numerically equal, so real-time doesn't strictly win either. Why this step? Correcting the earlier confusion: the inequality is strict; at equality it is simply false, which is exactly the "tie" — costs are identical, not "non-strict".
Tie-breaker. With costs equal, decide on staleness tolerance $S$ and latency SLA: staleness OK ( $Δ t \leq S$ ) → batch (simpler ops); freshness needed ( $Δ t > S$ ) → real-time. Why this step? Cell D is reserved for "the math is a tie; architecture preference wins."

Answer: Cost-neutral; pick by staleness/latency. Cell D.

Verify: $C_{batch} = 1 0^{- 4} /0.25 = 4 \times 1 0^{- 4}$ ; $C_{rt} = 1 0^{- 4} + 3 \times 1 0^{- 4} = 4 \times 1 0^{- 4}$ . Equal, and the strict test $0.25 > 0.25$ is False. ✓

Let's see the break-even geometry — it's the crossing point of two cost curves.

Figure s02 — Break-even curves (described so it stands alone). Horizontal axis is utilization $u$ ( $0$ to $1$ ); vertical axis is cost per useful prediction ($). The red curve is $C_{batch} = c / u$ , shooting toward infinity as $u \to 0$ (you waste almost everything). The dashed black horizontal line is the flat $C_{rt} = c + o$ . A red dot marks their crossing at $u = \frac{c}{c + o} = 0.25$ : to the left batch is dearer → real-time wins; to the right batch is cheaper → batch wins.

Worked example Example 7 — Cell G: infinite input space (can't precompute)

Setup: A search engine must rank results for arbitrary free-text queries. There are effectively infinitely many possible queries.

Forecast: What is $N$ , the number of predictions to precompute?

Count the input space. Distinct queries $\to \infty$ . To precompute all, $N \to \infty$ , so total batch cost $N c \to \infty$ . Why this step? Batch requires enumerating inputs ahead of time; an unbounded input space makes that impossible.
Utilization of a fully-precomputed table $\to 0$ (almost every precomputed query is never asked). Why this step? Even if you could enumerate, $u \to 0$ makes the cost rule fail catastrophically.

Answer: Real-time for the query-dependent part. Cell G — this is why search is inherently online.

Verify: As $N \to \infty$ with fixed $c > 0$ , $N c \to \infty$ (unbounded). Batch infeasible. ✓

Worked example Example 8 — Cell H: hybrid pipeline split

Setup: Same search engine. Document embeddings ( $N = 1 0^{7}$ docs, expensive, change rarely) vs matching this query (cheap, live). Batch part: $1 0^{7}$ docs $\times$ $2$ ms $= 2 \times 1 0^{4}$ s of nightly GPU. Real-time part: nearest-neighbor lookup at $8$ ms/query. Batch window $W = 6 h = 21, 600$ s.

Forecast: Which half goes where?

Classify by what changes slowly. Doc embeddings change only when documents change (rare) → batch. Why this step? The hybrid rule: precompute the stable, expensive part.
Classify the live part. The query is unknown until typed → real-time nearest-neighbor + re-rank. Why this step? Query-specific work has zero feasibility for precompute; it must be online — a batch window $W > 0$ containing the query never exists.
Check the batch half fits its window. Batch compute $= 1 0^{7} \times 2 \times 1 0^{- 3} = 2 \times 1 0^{4}$ s. Across $M = 100$ machines the per-machine time is $2 \times 1 0^{4} /100 = 200$ s, comfortably $< W = 21, 600$ s. Why this step? Feasibility check on the batch half — the stable part must finish inside the window, or the "fresh embeddings" promise breaks.

Answer: Hybrid. Embeddings batched (served via a Feature Store), matching served real-time. Cell H.

Verify: Batch compute $= 1 0^{7} \times 2 \times 1 0^{- 3} = 2 \times 1 0^{4}$ s. On $100$ machines: $2 \times 1 0^{4} /100 = 200$ s $< 21600$ s. ✓

Worked example Example 9 — Cell I: throughput vs latency, with queue stability (

Dynamic Batching) Setup: A real-time endpoint gets $λ = 2000$ requests/second. Processing them one-by-one costs $4$ ms each of GPU. Grouping $32$ requests into one batched forward pass takes $20$ ms total but must wait up to $8$ ms to fill the batch.

Forecast: Does batching 32 raise throughput enough to keep the queue stable? What's the latency cost?

One-at-a-time service rate: $μ_{1} = \frac{1}{4 ms} = 250$ predictions/second per GPU-slot. Why this step? Baseline: latency-optimal but throughput-poor.
Queue-stability check for single mode. Stability requires service rate $\geq$ arrival rate: $μ_{1} = 250 < λ = 2000$ . Unstable — the queue grows without bound, latency blows up. Why this step? The crucial edge case: at high RPS a slow single-item server cannot keep up, so raw latency-per-request is misleading. You must compare capacity to arrival rate.
Dynamic-batch service rate: $μ_{32} = \frac{32}{20 ms} = \frac{32}{0.020} = 1600$ predictions/second per slot. Why this step? Vectorized GPU work amortizes fixed overhead across $32$ items — this is why we batch.
Throughput gain & stability. Speed-up $= \frac{1600}{250} = 6.4 \times$ . Still $1600 < 2000$ , so one batched slot is still unstable; you need $⌈ 2000/1600 ⌉ = 2$ parallel slots ( $3200$ /s capacity) to stay stable — see Autoscaling. Why this step? Quantifies both the gain and how many replicas keep the queue bounded.
Latency cost: worst-case $= 8$ ms (fill) $+ 20$ ms (compute) $= 28$ ms vs $4$ ms single. We trade $+ 24$ ms latency for $6.4 \times$ throughput. Why this step? Latency and Throughput always trade — you must name the price and check the queue does not explode.

Answer: Batch of 32 gives $6.4 \times$ throughput at $+ 24$ ms worst-case latency, but needs $2$ replicas to stay stable at $2000$ RPS. Cell I.

Verify: $1/0.004 = 250$ ; $250 < 2000$ (unstable); $32/0.020 = 1600$ ; $1600/250 = 6.4$ ; $⌈ 2000/1600 ⌉ = 2$ ; latency $8 + 20 = 28$ ms. ✓

Picture the trade-off directly:

Figure s03 — Throughput vs latency (described so it stands alone). Horizontal axis lists the two serving modes; vertical axis is throughput (predictions/second). A short black bar marks "one-at-a-time" ( $250$ /s) and a tall red bar marks "dynamic batch (32)" ( $1600$ /s). Each bar is annotated with its per-request latency ( $4$ ms vs $28$ ms), so the tall red throughput bar visibly carries the higher latency label — the trade made explicit.

Worked example Example 10 — Cell J: exam twist — staleness beats cost (with a penalty function)

Setup: A dynamic-pricing model. Batch nightly would give $u = 0.9$ , $c = 1 \times 1 0^{- 4}$ , $o = 1 \times 1 0^{- 4}$ , so the threshold $= \frac{c}{c + o} = 0.5$ and $u = 0.9 > 0.5$ → cost says batch. BUT competitor prices change fast; a midnight price is $18$ h stale by evening, while the staleness tolerance is only $S = 2$ h.

Forecast: Cost says batch. Do you follow it?

Cost rule. Threshold $= \frac{1 0 ^{- 4}}{2 \times 1 0 ^{- 4}} = 0.5$ ; $u = 0.9 > 0.5$ (strict, true) → batch cheaper by pure compute. Why this step? Always run the numbers first — they genuinely favor batch here.
Staleness check. The staleness gap $Δ t = 18$ h vastly exceeds the tolerance $S = 2$ h, so "staleness OK" in Gate 2 is false. Why this step? Gate 2 has two conditions; passing cost is not enough if $Δ t > S$ .
Quantify the staleness loss. Model the expected loss from a stale prediction as a linear penalty $L (Δ t) = k \cdot Δ t,$ where $k$ = dollars of business loss per hour of staleness. Total real cost of batch $= C_{batch} + L (Δ t)$ . Why this step? The parent's "hidden cost of batch" made quantitative — cost-per-prediction alone ignores value destroyed by wrong prices.
Plug numbers. With $k = \$ 3\times10^{-4} $p er h o u r an d$ \Delta t = 18 $h,$ L = 3\times10^{-4}\times18 = $5.4\times10^{-3} $. T h e nba t c h^{'} s * t r u e * cos t$ \approx c/u + L = 1.11\times10^{-4} + 5.4\times10^{-3} \approx $5.5\times10^{-3} $, v er s u sr e a l - t im e$ C_{\text{rt}} = c+o = $2\times10^{-4}$. Why this step? Now the comparison is apples-to-apples: staleness turns "batch cheaper" into "batch far dearer once wrong answers are priced in".
Decide. $5.5 \times 1 0^{- 3} ≫ 2 \times 1 0^{- 4}$ → real-time (or short-window micro-batch) wins. Why this step? Cell J is the trap: cheap compute $\neq =$ correct answer once staleness is monetized.

Answer: Real-time despite compute favoring batch. Cell J. See Data Drift and Staleness.

Verify: threshold $= 0.5$ , $0.9 > 0.5$ True; $Δ t = 18 > S = 2$ ; $L = 3 \times 1 0^{- 4} \times 18 = 5.4 \times 1 0^{- 3}$ ; batch true cost $c / u + L = 1 0^{- 4} /0.9 + 5.4 \times 1 0^{- 3} \approx 5.51 \times 1 0^{- 3} > C_{rt} = 2 \times 1 0^{- 4}$ . ✓

Recall

Recall The whole matrix in one breath

Precompute infeasible (input born at deadline, no window $W > 0$ ) ::: real-time forced (Cell B/G). Feasible, high $u > \frac{c}{c + o}$ , staleness OK ( $Δ t \leq S$ ) ::: batch (Cell A). Feasible, low $u$ below threshold ::: real-time (Cell C). $u = \frac{c}{c + o}$ exactly ::: strict " $>$ " false → costs equal → decide by staleness/latency (Cell D). $o = 0$ ::: threshold $\to 1$ , real-time always wins on cost (Cell E). $o \to \infty$ ::: threshold $\to 0$ , batch almost always wins (Cell F). Cost says batch but $Δ t > S$ and penalty $L = k Δ t$ dominates ::: real-time anyway (Cell J). Expensive-stable + cheap-live ::: hybrid split (Cell H). Batching at high RPS ::: check service rate $\geq$ arrival rate for queue stability (Cell I).

Connections

Parent — Batch vs real-time inference
Model Serving — hosts the real-time endpoints in Cells B, C, G, I, J.
Feature Store — serves the batched embeddings of Cell H to live models.
Latency and Throughput — the trade quantified in Cell I.
Dynamic Batching — the compromise of Cell I.
Autoscaling — adds replicas for queue stability (Cell I) and absorbs overhead $o$ .
Data Drift and Staleness — the override force with penalty $L = k Δ t$ in Cell J.
Caching Strategies — how batch results in Cells A/H are stored and looked up.