Batch vs real-time inference: Exercises — AI-ML deep dive

Before we start, one tiny reminder of the quantities the whole page leans on, so no symbol is used before it's earned:

Common mistake Three edge cases hidden in the timing — read before Exercise 2.1

$u = 0$ (nobody ever uses a precomputed prediction): then $c / u = c /0 \to \infty$ . Batch cost blows up to infinity — you paid to compute for an audience that never showed. This is the extreme of "precomputing for users who never log in", so never batch a task with near-zero utilization.
$slack = 0$ (input arrives exactly at the deadline): there is no earlier moment to precompute in, so batch is not merely unwise — it is impossible. Real-time is forced. So the rule is precisely: $slack > 0 \Rightarrow$ batch possible; $slack = 0 \Rightarrow$ real-time forced.
$slack < 0$ ( $t_{d} < t_{a}$ : the deadline is before the input even arrives): you are already too late — no mode can produce a prediction in time, batch or real-time. The task is infeasible as stated; you must either relax the deadline ( $t_{d}$ later) or find an earlier signal ( $t_{a}$ sooner). Negative slack is a red flag that the problem, not the architecture, needs fixing.

The figure below plots exactly these two cost curves so you can see the crossover before computing it — the lavender batch curve $c / u$ shooting up as $u \to 0$ , the flat coral real-time line $c + o$ , and the mint break-even line where they meet. Its horizontal axis is utilization $u$ and its vertical axis is cost per useful prediction. Keep it in mind through all of Level 2 and 3.

Notice in the figure how the lavender curve rockets toward the top-left: that visual explosion is the $u \to 0$ edge case above. And everything to the right of the mint line (buttered region) is where batch wins — the exact inequality you'll derive in Exercise 2.2.

Level 1 — Recognition

Exercise 1.1

For each system, name the mode (batch or real-time) and the single clue that settles it: (a) A bank scores every account for churn risk every Sunday night and writes the score to a table. (b) A voice assistant transcribes the words you just spoke. (c) A photo app tags faces the instant you upload a new photo.

Recall Solution

(a) Batch — "every Sunday night" + "writes to a table" = scheduled, stored, looked up later. (b) Real-time — the input (your speech) arrives exactly when the answer is needed; slack $\approx 0$ . (c) Real-time — the photo you just uploaded didn't exist to precompute; on-demand.

Exercise 1.2

True or false, with one line of justification each: (a) Real-time inference optimises throughput above all else. (b) Staleness is a risk unique to batch inference. (c) "Batch size = 1" turns a batch pipeline into a real-time system.

Recall Solution

(a) False — real-time optimises latency (time for one request). Throughput (predictions/sec) is what batch chases. (b) True — batch computes early and consumes late, so the world can change in between. Real-time computes at consumption, so no staleness gap. (c) False — the model math becomes batch-size-1, but the system (live server, autoscaling, p99 SLA, cold starts) is entirely new. You choose an architecture, not a batch size.

Level 2 — Application

Exercise 2.1

A model costs $c = 4$ (compute units) per prediction. The per-request overhead is $o = 1$ . You run the batch job over $N = 1000$ candidates but only $u = 0.5$ get used. Compute the batch cost per useful prediction, the real-time cost per useful prediction, and say which mode is cheaper.

Recall Solution

Batch: total spend $= N c = 1000 \times 4 = 4000$ ; useful outputs $= u N = 500$ ; so $C_{batch} = \frac{N c}{u N} = \frac{4000}{500} = 8$ , which equals $\frac{c}{u} = \frac{4}{0.5} = 8$ (the $N$ cancels, as promised). Real-time: $C_{rt} = c + o = 4 + 1 = 5$ . Since $5 < 8$ , real-time is cheaper. Half of what you precompute is thrown away, doubling the effective cost — the small overhead of $1$ is a bargain by comparison.

Exercise 2.2

Same $c = 4$ , $o = 1$ . Find the exact utilization $u$ at which batch and real-time cost the same, then state the rule for when batch wins.

Recall Solution

Set costs equal: $\frac{c}{u} = c + o$ . WHY isolate $u$ ? We want the utilization threshold — the value of $u$ that makes the two modes tie — so we solve for $u$ alone. Two domain facts make this legal: $c + o > 0$ (compute plus overhead is always a positive price), so we may divide by it; and $u > 0$ (you use some predictions, else batch is the $u = 0$ infinity case), so we may multiply both sides by $u$ without flipping or dividing by zero. Then $c = u (c + o) \Rightarrow u = \frac{c}{c + o} = \frac{4}{5} = 0.8$ . Batch wins when $u > 0.8$ (you use more than 80% of what you precompute). Below that, wasted compute makes batch dearer than paying the overhead $o$ on demand.

Exercise 2.3

An input arrives at $t_{a} = 12:00$ and the answer is needed by $t_{d} = 18:00$ the same day. State the slack in hours, and whether precomputing (batch) is possible. Then say what happens in the boundary case $slack = 0$ .

Recall Solution

$slack = t_{d} - t_{a} = 18 : 00 - 12 : 00 = 6$ hours $> 0$ , so precomputation is possible — there is a window to do the work early. But possible $\neq =$ wise: if the underlying data can change within those 6 hours (price drop, new purchase), the precomputed answer goes stale. Positive slack unlocks batch; low staleness cost justifies it. Boundary $slack = 0$ : input and deadline coincide, so there is no earlier moment to precompute in — batch is impossible, and real-time is forced regardless of any cost analysis (this is exactly the fraud case in Level 5). (And if $slack < 0$ , the deadline is already past when the input arrives — no mode can help; the problem itself must be re-scoped.)

Level 3 — Analysis

Exercise 3.1

A model is cheap: $c = 2$ , overhead $o = 8$ (heavy network + serialization). Find the break-even utilization. What does the large $o / c$ ratio tell you about which mode this workload leans toward?

Recall Solution

$u^{⋆} = \frac{c}{c + o} = \frac{2}{10} = 0.2$ . Batch wins whenever $u > 0.2$ — a very low bar. Here $\frac{o}{c} = 4$ is large: each on-demand call carries heavy overhead, so real-time is expensive per request. This workload leans batch: even at modest utilization, precomputing beats paying the fat overhead repeatedly. (Contrast Exercise 2.2 where tiny overhead pushed break-even up to 0.8, favouring real-time.)

Exercise 3.2

You run real-time with dynamic batching: the server waits up to $w = 8$ ms to collect concurrent requests, then processes them together. A request arrives $3$ ms into a collection window. Its latency budget is $50$ ms and the forward pass takes $12$ ms. (a) What is this request's worst-case added wait from batching? (b) Does it fit the budget assuming $5$ ms network overhead?

Recall Solution

(a) The window closes $8 - 3 = 5$ ms after this request arrives (worst case it arrived just as the window opened, giving the full $8$ ms; here it's $5$ ms of remaining wait). (b) Total = batching wait $+$ forward pass $+$ network $= 5 + 12 + 5 = 22$ ms $< 50$ ms. Fits, with $28$ ms of headroom. Dynamic batching buys throughput by spending a little of that headroom — see Dynamic Batching.

Exercise 3.3

Two designs serve the same feature. Design A: real-time, $c = 3$ , $o = 2$ , $u \approx 1$ . Design B: batch, $c = 3$ , $u = 0.9$ , plus a fixed storage cost $s = 0.5$ per useful prediction to hold results. Which is cheaper per useful prediction?

Recall Solution

A (real-time): $c + o = 3 + 2 = 5$ . B (batch): $\frac{c}{u} + s = \frac{3}{0.9} + 0.5 = 3.333 \dots + 0.5 = 3.833 \dots$ $3.83 < 5$ , so batch (B) is cheaper. High utilization ( $0.9$ ) keeps waste low and the storage add-on $s$ doesn't erase the win. Storage is batch's often-forgotten cost — see Caching Strategies.

Level 4 — Synthesis

Exercise 4.1

Design the inference architecture for a search engine. Documents number in the billions and change slowly; each user query is unique and must return in $< 100$ ms. Split the pipeline into a batch part and a real-time part, and justify each split by what changes slowly vs what depends on the live request.

Recall Solution

Batch part (nightly / on doc update): compute an embedding (a vector representation) for every document. Documents are stable and enormous in number — high slack, high reuse — so precompute and store the vectors in a Feature Store / vector index. Real-time part (per query): embed this query, do a nearest-neighbour lookup against the precomputed index, then a light re-rank. This depends on the live request (the query text didn't exist until now) and must fit the $100$ ms budget. Justification: split by rate of change. The expensive, stable half is batched; the cheap, query-specific half is served live. This is the hybrid pattern from the parent's Worked Example 3.

Exercise 4.2

Your real-time endpoint sees a daily traffic spike from $100$ to $1000$ requests/sec at 9 a.m. Each pod (server replica) handles $150$ req/sec. Describe the scaling plan, and explain the cold-start risk and one mitigation.

Recall Solution

Replica count needed: at peak, $⌈ 1000/150 ⌉ = ⌈ 6.67 ⌉ = 7$ pods; off-peak, $⌈ 100/150 ⌉ = 1$ pod. Use Autoscaling to scale between 1 and 7 based on load. Cold-start risk: a newly spun-up pod must load model weights into memory before serving — during that gap it cannot answer, so requests queue or fail, spiking p99 latency exactly when traffic surges. Mitigation: keep a warm pool (a couple of pre-loaded idle pods) or pre-scale on a schedule ahead of the known 9 a.m. spike, so pods are ready before the wave hits. See Latency and Throughput.

Level 5 — Mastery

Exercise 5.1

A fraud model runs real-time at $c = 6$ , $o = 3$ , $u \approx 1$ . A colleague proposes a "cost-saving" batch job that precomputes a risk score per card each morning ( $u = 0.7$ , storage $s = 0.4$ per useful prediction). (a) Compare pure costs. (b) Give the decisive reason batch fails here that has nothing to do with cost.

Recall Solution

(a) Real-time: $c + o = 6 + 3 = 9$ . Batch: $\frac{c}{u} + s = \frac{6}{0.7} + 0.4 = 8.571 \dots + 0.4 = 8.971 \dots \approx 8.97$ . Batch is marginally cheaper ( $8.97 < 9$ ) — tempting! (b) But cost is irrelevant here. A fraud decision depends on this swipe's device, location, amount, and time — a feature combination that did not exist until the swipe happened. A morning-computed "risk score per card" cannot see the live transaction, so it answers the wrong question. Slack is $\approx 0$ ; staleness is unacceptable. Real-time is forced regardless of the penny it saves.

Exercise 5.2

Build a single decision procedure (a short flowchart in words) that takes any new prediction task and outputs batch, real-time, or hybrid. Then run it on: recommending articles that stay relevant for a day, with $u = 0.65$ , $c = 5$ , $o = 4$ .

Recall Solution

Procedure (in order — the first "yes" that forces a mode wins):

Is the deadline even reachable — is $slack \geq 0$ ? No ( $t_{d} < t_{a}$ ) → task infeasible; re-scope the deadline or find an earlier signal.
Does the input exist before the deadline with room to precompute — is $slack > 0$ ? No ( $slack = 0$ ) → real-time (forced).
Is staleness over the slack window unacceptable? Yes → real-time (forced).
Can the pipeline be split into a slow-changing part and a live-request part? Yes → hybrid.
Otherwise compare cost: batch if $u > \frac{c}{c + o}$ , else real-time.

Run on the article task: $slack \geq 0$ ✓ and $slack > 0$ (recs valid for a day) ✓; staleness over a day is acceptable ✓ (not forced to real-time); not obviously splittable into slow vs live parts → fall through to the cost step. Break-even $u^{⋆} = \frac{c}{c + o} = \frac{5}{9} = 0.5 \overline{5} \approx 0.556$ . Actual $u = 0.65 > 0.556$ , so batch wins. Precompute the day's recommendations and cache them (see Caching Strategies).

Exercise 5.3

A hybrid search system precomputes doc embeddings in batch every 24 h. Freshly published documents therefore wait up to a day to become searchable. Propose a fix that keeps the batch efficiency but bounds worst-case staleness to under 1 hour, and name the drift concept at play.

Recall Solution

Fix: add an incremental / mini-batch stream — a small job running every, say, 15 minutes that embeds only newly published documents and upserts them into the same index. The heavy nightly batch still handles the bulk (efficient), while the mini-batch bounds worst-case staleness to $\leq 15$ min $< 1$ h. Concept: this is managing Data Drift and Staleness — the gap between when a prediction (here, an embedding) is computed and when it's consumed. Bounding that gap bounds staleness cost.

Recall One-line ladder recap

L1 spot the mode from clues → L2 plug the cost formulas → L3 include hidden storage/overhead → L4 design & scale the system → L5 let forcing constraints trump cost, and bound staleness in hybrids.

Connections

Parent: Batch vs real-time inference
Model Serving, Feature Store, Latency and Throughput, Autoscaling, Data Drift and Staleness, Dynamic Batching, Caching Strategies