Batch vs real-time inference: Question bank — AI-ML deep dive

Before you start, two symbols we keep reusing — both defined in the parent, restated in plain words here so no line assumes them:

True or false — justify

Batch inference is always cheaper than real-time because it uses GPUs efficiently.

False — batch is cheap per prediction only when utilization

u

is high; if you precompute for millions who never log in, you paid

N c

but only

u N

got used, so cost per useful answer

c / u

can exceed real-time's

c + o

"Real-time inference = batch with batch size 1" is just a config change.

False — the model math is identical, but real-time forces a live server, load balancing, autoscaling, request queues, and p99 latency SLAs; you're picking an architecture, not a batch size.

If slack is zero, you might still choose batch to save money.

False — zero slack means the input doesn't exist until the moment you need the answer, so there is literally no earlier moment to precompute in; cost is irrelevant, real-time is forced.

A batch prediction and the equivalent real-time prediction return the exact same number.

True at the instant of computation — same weights, same forward pass — but by the time a batch answer is consumed, the world may have changed, so the number can be right yet stale.

Dynamic batching turns a real-time system into a batch system.

False — dynamic batching waits only a few milliseconds to group concurrent live requests for GPU efficiency; it still serves each on-demand within a latency budget, so it stays real-time.

Higher throughput always means lower latency.

False — they trade off; batch groups inputs to raise throughput (predictions/sec) but each input waits for the batch to fill, which raises the latency of any single prediction.

A model with tiny per-request overhead

o

pushes you toward real-time.

True — as

o \to 0

the break-even threshold

\frac{c}{c + o} \to 1

, so batch only wins at near-perfect utilization; otherwise cheap on-demand serving beats wasting compute.

Spot the error

"We serve fraud detection in batch every hour, so latency is basically zero."

The error: a fraud decision must use this swipe's features, which don't exist until the swipe; an hourly batch can't score a transaction that hasn't happened, so this design can't work at all, not just "slowly."

"Our utilization is

u = 0.3

and overhead is negligible, so batch is the cheaper choice."

The error: with negligible

o

, the threshold

\frac{c}{c + o} \approx 1

, and

u = 0.3 < 1

, so batch is wasteful — real-time is cheaper; low utilization is exactly when batch loses.

"Batch has no downside once we accept the nightly compute cost."

The error: batch's fundamental weakness is staleness — an answer computed at midnight and consumed at 6 PM can be wrong if the world (price, purchase, location) changed in between.

"We can precompute recommendations for every possible query, so real-time is never needed."

The error: the input space explodes — the exact feature combination (this query, this device, this moment) is effectively infinite; you cannot precompute what doesn't yet exist.

"Search ranking must be entirely real-time because queries are live."

The error: only the query-specific part must be live; the expensive, slowly-changing part (document embeddings) is precomputed in batch, then matched real-time — a hybrid, not all-or-nothing.

"Adding a warm idle server has no cost, so real-time overhead

o

is zero."

The error: keeping a server warm and idle is part of

o

— you pay for reserved capacity even when no request arrives, on top of network and serialization cost.

"A Feature Store means we're doing pure real-time inference."

The error: a feature store bridges both worlds — it serves batch-precomputed features into real-time models, so its presence signals a hybrid, not a mode.

Why questions

Why does the parent say "every prediction has two clocks"?

Because the choice depends not on the model but on when the input is known (

t_{a}

) versus when the answer is due (

t_{d}

); their gap, the slack, decides whether precomputation is even possible.

Why is batch cost

\frac{c}{u}

and not

c

Because you pay compute

c

for all

N

candidates (

N c

total) but only

u N

get consumed, so spreading the bill over useful predictions gives

\frac{N c}{u N} = \frac{c}{u}

Why does real-time assume

u \approx 1

Because you only compute when a request actually arrives, so essentially every prediction is used — there's no precomputing for consumers who never show up.

Why can staleness make batch "silently wrong"?

The prediction still returns cleanly with no error, so nothing flags it — but its inputs are from midnight while reality moved on, so it's confidently outdated rather than obviously broken.

Why does high utilization and an expensive model favour batch?

High

u

means little waste, and an expensive

c

makes recomputing per request costly; precomputing once overnight and caching the result amortizes that expense across many cheap lookups.

Why split a pipeline into batch and real-time stages instead of picking one?

Because different stages have different clocks — the slow, stable, expensive part (e.g. embeddings) has huge slack and belongs in batch, while the query-dependent part has zero slack and must be live.

Edge cases

What happens to the decision when slack is large but staleness tolerance is tiny (answer must be fresh to the second)?

Large slack allows batch, but zero staleness tolerance forbids it — you must recompute at use-time, so you're pushed to real-time (or a very short refresh interval) despite the slack.

Utilization is exactly at the break-even

u = \frac{c}{c + o}

— which mode?

Cost is a tie (

\frac{c}{u} = c + o

), so cost no longer decides; break the tie on secondary factors — staleness tolerance, latency SLA, and operational simplicity.

What if

u = 1

(every precomputed prediction is consumed) — is batch automatically best?

Not automatically;

u = 1

makes batch cost-optimal only if slack is positive and staleness is acceptable — a live-input, zero-slack task still can't be batched no matter how high

u

is.

Overhead

o

is huge (heavyweight cold-start server). What does the threshold do?

The threshold

\frac{c}{c + o}

drops toward zero, so even low utilization makes batch cheaper — heavy per-request overhead punishes on-demand serving.

The input space is small and fully enumerable (say, 50 possible inputs). Does the "can't precompute" rule still apply?

No — if every possible input is known and fixed, you can precompute all 50 answers and just look them up; the "can't precompute" argument only bites when inputs are unknown until the live event.

A request arrives for a user whose batch prediction was never computed (new signup at 3 PM). What breaks?

The batch table has no row for them — a cache miss — so the system must fall back to real-time inference or a default, revealing that pure batch can't cover freshly-arrived inputs.

Compute cost

c

is essentially free (a tiny model). Which mode wins on cost?

The threshold

\frac{c}{c + o} \to 0

, so almost any utilization favours batch on paper — but with

c

tiny, real-time is also cheap and avoids staleness, so freshness usually decides instead of cost.

Recall One-line self-test

Cover the answers. If you can state why (utilization, slack, staleness, overhead) and not just which, you've internalized the topic. Which single quantity, if zero, ends the debate immediately? ::: Slack — zero slack means the input arrives exactly when the answer is due, so precomputation is impossible and real-time is forced.

Connections

Batch vs Real-time Inference — the parent topic these traps stress-test.
Latency and Throughput — the trade-off behind half the false statements here.
Data Drift and Staleness — why batch answers silently rot.
Dynamic Batching — the real-time-that-looks-like-batch middle ground.
Feature Store — the hybrid bridge that trips the "pure mode" traps.
Caching Strategies — how precomputed batch answers get stored and missed.
Model Serving — the live infra whose overhead is the $o$ in every cost line.