Intuition The one idea this whole topic rests onA model is just a fixed sequence of arithmetic steps turning input numbers into output numbers — and that sequence can be written down as a graph (ONNX) and run anywhere , including on the tiny chip inside your phone. Everything else — latency budgets, int8 quantization, execution providers — is just making that graph run fast and small enough to live at the edge, where the data is born .
This page builds every symbol the parent note uses, from zero, before you meet it in a formula. If a smart 12-year-old reads line one first, they should never hit a symbol they haven't been shown a picture of.
Intuition Cloud vs edge, as a pictureImagine a photo taken on your phone. In the cloud path the photo travels across the internet to a big computer, gets an answer, and the answer travels back. In the edge path the phone answers by itself . The whole topic is about when — and how — the second path wins.
A small computer at the place where data appears : a phone, a security camera, a car sensor. It has a weak chip and often no reliable internet. Contrast with a server in a data centre .
Before any latency formula, three plain quantities.
A byte is one unit of computer memory (8 tiny on/off switches). A number stored as "float32" takes 4 bytes ; the same number squeezed into "int8" takes 1 byte . Picture four boxes vs one box.
B B bytes B t ≈ B bytes . Why we need B the parent's transfer terms t up , t down directly from this — bigger data or thinner pipe ⇒ longer wait.
Every symbol with a t a duration in seconds . The subscript tells you which duration.
Symbol
Plain meaning
Picture
t up time to send data up to cloud
photo climbing the pipe
t down time for answer to come down
answer sliding back
t queue wait in line before the server serves you
people ahead of you at a counter
t infer cloud compute time on the big chip
fast machine thinking
t infer edge compute time on the tiny chip
slow machine thinking
Inference = running an already-trained model on new input to get a prediction. (Not training — no learning happens; just the arithmetic runs forward once.)
Definition Round-trip time (RTT)The total up + down travel time for one exchange, ignoring compute. Picture a ball thrown to a friend and back.
Intuition Reading the parent's edge-wins inequalityThe parent shows edge wins when
penalty of the smaller chip t infer edge − t infer cloud < network + queue cost t up + t down + t queue . left is "how much slower the tiny chip is", the right is "everything you avoid by not phoning the cloud". Edge wins when avoiding the trip is worth more than the slower chip.
Worked example Plug in real numbersCloud: t up = 40 ms t infer cloud = 5 ms t down = 10 ms t queue = 20 ms t infer edge = 45 ms
Cloud total = 40 + 5 + 10 + 20 = 75 ms
Edge total = 45 ms
45 < 75 edge wins , even though its chip is slower (45 > 5
This is the heart. If you get this picture, ONNX becomes obvious.
Definition Operator (op) / nodeAn operator is one arithmetic step: multiply two number-grids (MatMul), slide a filter over an image (Conv), or set negatives to zero (Relu). A node is one such op sitting in the graph. See Computational Graphs .
A tensor is just a box of numbers with a shape. A single number is shape (); a list is (n); a colour image is (3, H, W) = 3 colour grids of height H W (1, 3, 224, 224) in the parent means: 1 image, 3 colours, 224×224 pixels.
Definition Weights / initializersThe weights are the fixed numbers the model learned during training. In an ONNX file they are stored as initializers — numbers baked in, never changing at inference time.
Intuition A model = a graph of ops flowing tensorsPicture arrows carrying tensors into boxes (ops), which output new tensors into the next boxes. That directed picture is the model. ONNX is nothing more than writing that picture down in a standard file.
Definition Input / output namesEach entry arrow and exit arrow gets a label, e.g. "img" in, "logits" out. Why: when you run the model you must say which numbers go into which entry arrow — you address them by name.
The raw output numbers before turning them into probabilities. Bigger logit = model more confident in that class.
The program that actually executes the graph — reads each node and does the arithmetic. ONNX Runtime (ORT) is one such program.
Definition Execution Provider (EP)An EP is the specific backend ORT uses to compute a node: plain CPU, a GPU via CUDA or TensorRT , or a phone chip via CoreML / NNAPI . Picture the same recipe cooked on different stoves.
Definition Opset (operator set version)A version number saying which edition of the operators the graph uses (opset_version=17). Why it matters: a runtime only understands editions up to some number. Ask for a newer edition than the runtime knows ⇒ it refuses to load.
Common mistake Confusing "runtime" with "framework"Feels right: you trained in PyTorch, so PyTorch runs it.
Truth: the framework (PyTorch/TF) trains; the runtime (ORT) runs the exported graph — possibly on a device that has no PyTorch at all. That decoupling is the whole point of ONNX.
The parent's int8 derivation uses four letters. Here is each, from zero.
Definition The four quantization symbols
r real value (the true float number, e.g. a weight like 0.5 q quantized integer that stands in for r 64).S scale : how many real units one integer step is worth. Picture the spacing of tick marks on a ruler.Z zero-point : the integer that represents real 0
Intuition The picture behindr = S ( q − Z ) [ r m i n , r m a x ] [ q m i n , q m a x ] S Z 0
The function that snaps a decimal to its nearest whole number: round ( 63.75 ) = 64 Why it appears: integers can't hold 63.75, so we must snap — and that snap is exactly where quantization error is born (∣ r ^ − r ∣ ≤ S /2
Definition Calibration & QAT (named, so the parent's fixes make sense)
Calibration dataset : a handful of real inputs used to measure the true [ r m i n , r m a x ] S Quantization-aware training (QAT) : training that pretends to round, so weights adapt to it. Deep-dived in Model Quantization and Pruning .
N M O ( ⋅ ) N M O ( expression ) O ( N ⋅ M ) pair ; O ( N + M )
Recall Why isN + M N ⋅ M N = 5 , M = 5 = 25 = 10 N , M
time symbols t up t down t queue
Cover the right side; can you answer before revealing?
A byte, and how many a float32 vs int8 weight takes ::: 1 byte = 8 switches; float32 = 4 bytes, int8 = 1 byte (4× smaller).
What B t ≈ bytes / B
What each t
What "inference" means vs "training" ::: running a trained model forward on new input; no learning happens.
What a tensor is and what shape (1,3,224,224) encodes ::: a numbered box of numbers; 1 image, 3 colours, 224×224 pixels.
What an operator/node is, with two examples ::: one arithmetic step in the graph; e.g. MatMul, Conv, Relu.
What weights/initializers are ::: the fixed learned numbers baked into the graph.
What a computation graph is ::: ops as boxes, tensors as arrows, flowing input to output.
What ONNX stores ::: the graph = nodes + operators + initializers (weights) + typed inputs/outputs.
Difference between a framework and a runtime ::: framework trains (PyTorch/TF); runtime executes the graph (ONNX Runtime).
What an Execution Provider is ::: the backend that actually computes a node (CPU, CUDA, TensorRT, CoreML).
What an opset version controls ::: which edition of operators the graph uses; runtime must support it.
The four quantization symbols r , q , S , Z
Why round() introduces error and its bound ::: integers can't hold decimals; error ≤ S /2
Why O ( N ⋅ M ) O ( N + M )
Ready? Then head back to the parent topic .
