Foundations — Batch vs real-time inference

This page assumes you know nothing. We will build every letter the parent note uses — $t_a$, $t_d$, slack, $N$, $c$, $u$, $o$, latency, throughput — from pictures, one at a time, each earned before the next.

0. What is "inference"?

We need this word first because everything else — batch, real-time, cost, latency — is just a way of organising when and how many presses of this button happen.

1. The two clocks: $t_a$ and $t_d$

Every prediction lives on a timeline with two special moments.

Look at the figure. Time flows left → right. The blue dot is $t_a$ (input born). The pink dot is $t_d$ (answer due). The prediction must be finished somewhere in the gap between them.

• Why two clocks and not one? Because the whole topic is about the distance between them. If you only had one moment, there'd be no choice to make.

2. Slack — the gap that decides everything

Why does the topic need this single number? Because it is the master switch:

• Large slack (yellow bar long): the input showed up long before the answer is due → you have time to spare → you can compute early → batch is possible.
• Tiny slack (dots almost touching): the input arrives the instant the answer is needed → there is no "earlier" moment → you are forced into real-time.

3. Counting the work: $N$

Picture: $N$ is the number of buttons pressed on our vending machine in one day. It's a plain count — no units, just "how many."

Why the topic needs it: cost scales with $N$. Doing 10 predictions cheaply and doing 200 million predictions are different worlds, and $N$ is the dial between them.

4. The price of one press: $c$

We need $c$ because it lets us compare "wasting compute on unused predictions" against "paying extra to serve on demand." Without a per-press price, no cost comparison is possible.

5. How much of it is wasted: $u$

Here is the subtle one. In batch, you often predict for everyone, but only some people actually use their answer.

Look at the bar in the figure: the whole bar is all $N$ predictions you paid for; the blue slice ($uN$ of them) is the part that did useful work; the grey slice was computed and thrown away.

• $u$ close to $1$: almost everything you computed was used → batch is efficient.
• $u$ close to $0$: you precomputed mountains of answers nobody read → batch is wasteful.

Why the topic needs it: this is the single number that penalises "predict for everyone even if they never log in." It turns wasted work into a cost you can put on a scale.

6. The cost of staying ready: $o$

Real-time doesn't waste predictions ($u\approx 1$, you only compute when asked). But it pays a different tax.

Why the topic needs it: real-time's total per-useful-prediction cost is $c+o$. Batch's is $c/u$. The whole decision rule is a fight between these two, so both $o$ and $u$ must exist as named quantities.

7. Latency vs throughput — two different stopwatches

These two words sound similar and get confused constantly. They measure different things.

The figure shows the difference with a checkout analogy. Top lane: one customer, timed from entry to exit — that arrow is latency. Bottom: total customers leaving per minute across all lanes — that count is throughput.

8. Staleness — why an early answer can be wrong

The topic needs this word because it is batch's fundamental weakness — the hidden cost that balances real-time's serving-infra cost.

The prerequisite map

Every arrow says "you need this before you can reason about that." Slack decides whether you have a choice; the cost model and the latency/throughput trade-off decide which choice; staleness is the tax on choosing batch.

Equipment checklist

Cover the right side. If you can answer each, you're ready for the parent note.