Edge deployment and ONNX: Worked examples — AI-ML deep dive

We reuse only two engines from the parent:

Latency: edge wins when $t_{infer}^{edge} < t_{up} + t_{infer}^{cloud} + t_{down} + t_{queue}$ .
Quantization: the affine map $r = S (q - Z)$ with $S = \frac{r _{m a x} - r _{m i n}}{q _{m a x} - q _{m i n}}$ , $Z = q_{m i n} - \frac{r _{m i n}}{S}$ , and round-trip error $∣ \overset{r}{^} - r ∣ \leq S /2$ .

The scenario matrix

Every worked example below is tagged with the cell it covers.

Cell	Scenario class	What can go wrong
A	Symmetric range $[- a, a]$	zero-point should land on $0$
B	Asymmetric / positive-only range $[0, r_{m a x}]$ (post-ReLU)	zero-point shifts off centre
C	Negative-only range $[r_{m i n}, 0]$	sign of $Z$ , does real-0 still map to an integer?
D	Value outside the range (outlier)	saturation / clamping at $q_{m i n}, q_{m a x}$
E	Degenerate range $r_{m i n} = r_{m a x}$	division by zero in $S$
F	Error bound as a limit	shrink range → shrink error
G	Latency break-even (real-world word problem)	edge-vs-cloud decision
H	Latency at the boundary (tie)	equality case, degenerate bandwidth
I	Exam-style twist: 4× size vs accuracy trade	reasoning, not just plugging

Example 1 — Cell A: symmetric range, the "textbook" case

Worked example Quantize

$r = 0.5$ with range $[- 1, 1]$ into int8 $[- 128, 127]$ . Forecast: guess the integer $q$ before reading. Closer to $0$ or closer to $127$ ?

Step 1. Compute the scale $S = \frac{1 - ( - 1 )}{127 - ( - 128 )} = \frac{2}{255} \approx 0.007843$ . Why this step? $S$ is "how many real units per integer tick" — we need it before anything else.

Step 2. Zero-point $Z = - 128 - \frac{- 1}{0.007843} = - 128 + 127.5 = - 0.5 \to round to 0$ . Why this step? $Z$ is the integer that represents real $0$ ; for a symmetric range it should sit near $0$ .

Step 3. Quantize: $q = round (0.5/0.007843) + 0 = round (63.75) = 64$ . Why this step? apply $q = round (r / S) + Z$ — the forward map.

Step 4. Dequantize back: $\overset{r}{^} = 0.007843 \times 64 = 0.5019$ . Why this step? to measure the rounding damage.

Verify: error $= ∣0.5019 - 0.5∣ = 0.0019 \leq S /2 = 0.0039$ . Inside the guaranteed bound. ✓ Related tools: Model Quantization and Pruning.

Example 2 — Cell B: positive-only range (post-ReLU activations)

Example 3 — Cell C: negative-only range

Example 4 — Cell D: an outlier OUTSIDE the range (saturation)

Worked example Range calibrated to

$[- 1, 1]$ , int8 $[- 128, 127]$ . A rare outlier $r = 3.0$ arrives. What happens? Forecast: does $q$ keep climbing past $127$ , or stop?

Step 1. $S = 2/255 \approx 0.007843$ , $Z = 0$ (same as Example 1). Why this step? calibration was done earlier; new inputs reuse the frozen $S, Z$ .

Step 2. Raw forward map: $q_{raw} = round (3.0/0.007843) + 0 = round (382.5) = 382$ . Why this step? the formula doesn't know about limits — it just computes.

Step 3. Clamp to the representable range: $q = min (127, max (- 128, 382)) = 127$ . Why this step? an 8-bit integer physically cannot exceed $127$ ; hardware saturates. Look at the red flat cap in the figure — every value above $r_{m a x}$ collapses onto the same code.

Step 4. $\overset{r}{^} = 0.007843 \times (127 - 0) = 0.9961$ . Why this step? to see the huge error the outlier causes.

Verify: error $= ∣0.9961 - 3.0∣ = 2.0039 ≫ S /2$ . The bound $S /2$ only holds inside the range — this is exactly why calibration and QAT matter (parent's "quantizing is free accuracy" mistake).

Example 5 — Cell E: degenerate range (division by zero)

Worked example A "dead" channel has all weights equal to

$2.0$ , so $r_{m i n} = r_{m a x} = 2.0$ . Quantize it. Forecast: what does $S$ do when the numerator is $0$ ?

Step 1. $S = \frac{r _{m a x} - r _{m i n}}{q _{m a x} - q _{m i n}} = \frac{2.0 - 2.0}{255} = \frac{0}{255} = 0$ . Why this step? a zero-width range needs zero real units per tick.

Step 2. $Z = q_{m i n} - r_{m i n} / S$ → division by zero. The formula breaks. Why this step? to expose the degeneracy rather than crash silently.

Step 3. Fix / convention: when $r_{m i n} = r_{m a x}$ , set $S = 1$ (or any nonzero) and store the constant directly; every input dequantizes to the single value $2.0$ . Why this step? real quantization libraries special-case zero-width ranges to avoid NaN.

Verify: with the fix, any $q$ gives $\overset{r}{^} = 2.0$ exactly (the channel is constant), error $= 0$ . ✓ Degenerate inputs must be caught, never fed to the raw formula.

Example 6 — Cell F: the error bound as a LIMIT

Worked example Same value

$r = 0.5$ , but we tighten the range from $[- 1, 1]$ to $[- 0.5, 0.6]$ (per-channel calibration). How much does the worst-case error shrink? Forecast: halving-ish the range — does the error bound roughly halve too?

Step 1. Old bound: $S_{old} = 2/255$ , so $S_{old} /2 = 1/255 \approx 0.003922$ . Why this step? baseline worst case.

Step 2. New scale: $S_{new} = \frac{0.6 - ( - 0.5 )}{255} = \frac{1.1}{255} \approx 0.004314$ . Why this step? tighter real span ⇒ smaller $S$ .

Step 3. New bound $S_{new} /2 = 0.55/255 \approx 0.002157$ . Why this step? error scales linearly with $S$ ; $∣ \overset{r}{^} - r ∣ \leq S /2$ .

Step 4. Ratio: $\frac{S _{new} /2}{S _{old} /2} = \frac{0.55}{1.0} = 0.55$ . Why this step? the limiting behaviour — as range span $\to 0$ , worst-case error $\to 0$ proportionally.

Verify: $0.002157 \approx 0.55 \times 0.003922$ . ✓ This is the mathematical reason per-channel ranges beat one global range.

Example 7 — Cell G: latency break-even (real-world word problem)

Worked example A phone must classify a

$150 KB$ image. Upload+download bandwidth $B = 1 MB/s$ (so $\approx 1000 KB/s$ ). Cloud inference $= 10 ms$ , queue $= 30 ms$ . The phone's own model runs in $t_{infer}^{edge} = 120 ms$ . Edge or cloud? Forecast: cloud's chip is 12× faster at compute — but does that win overall?

Step 1. Transfer time each way: $t_{up} = t_{down} = \frac{150 KB}{1000 KB/s} = 0.15 s = 150 ms$ . Why this step? $t \approx bytes / B$ — the parent's transfer model. (Result label ≈ 0.5 KB, i.e. logits, is tiny; we keep 150 ms as a conservative symmetric estimate.)

Step 2. $T_{cloud} = t_{up} + t_{infer}^{cloud} + t_{down} + t_{queue} = 150 + 10 + 150 + 30 = 340 ms$ . Why this step? sum every term the cloud path pays.

Step 3. $T_{edge} = 120 ms$ . Why this step? edge pays only its own compute — no transfer, no queue (look at the two stacked bars in the figure).

Step 4. Compare: $120 < 340$ ⇒ edge wins by $220 ms$ . Why this step? the decision rule: edge wins when network+queue ( $150 + 150 + 30 = 330$ ) exceeds the compute penalty ( $120 - 10 = 110$ ). $330 > 110$ . ✓

Verify: compute penalty $110 ms <$ network+queue $330 ms$ , consistent with $T_{edge} < T_{cloud}$ . ✓ See Model Serving and Inference Latency.

Example 8 — Cell H: the exact tie (boundary case)

Worked example Same numbers as Example 7, but bandwidth drops so that we ask:

at what $t_{infer}^{edge}$ do edge and cloud tie, and what does the tie mean? Forecast: the tie edge-time equals cloud's total, right?

Step 1. Set $T_{edge} = T_{cloud}$ : $t^{⋆} = 150 + 10 + 150 + 30 = 340 ms$ . Why this step? the break-even is defined by equality.

Step 2. Interpretation: if the edge model took exactly $340 ms$ , both paths finish together — pick edge for privacy/offline as a tiebreaker. Why this step? equality cases still need a decision; non-latency forces break the tie.

Step 3. Degenerate sub-case: if bandwidth $B \to \infty$ , then $t_{up}, t_{down} \to 0$ and $T_{cloud} \to 10 + 30 = 40 ms$ ; now edge ( $120$ ) loses. Why this step? the limiting behaviour — infinite bandwidth kills the edge advantage, leaving only queue+compute.

Verify: at $B \to \infty$ , $T_{cloud} = 40 ms < 120 ms$ , so cloud wins. The crossover bandwidth is exactly where $150 KB / B \times 2 = 340 - 40 = 300 ms$ ⇒ $B = 1000 KB/s$ , matching Example 7. ✓

Example 9 — Cell I: exam twist (size vs accuracy reasoning)

Worked example "Int8 quantization makes a

$40 MB$ float32 model $4 \times$ smaller. A teammate claims that means exactly $4 \times$ faster inference AND identical accuracy. Which claim(s) are wrong, and what is the true new size?" Forecast: how many of the two claims survive?

Step 1. New size $= 40 MB /4 = 10 MB$ (float32 = 4 bytes → int8 = 1 byte). Why this step? size scales exactly with bytes-per-weight; this part is true.

Step 2. " $4 \times$ faster" — false in general. Speedup depends on whether the chip has fast int8 kernels and whether the model is memory-bound or compute-bound; often $2$ – $3 \times$ , sometimes near $1 \times$ . Why this step? size ratio ≠ latency ratio (see TensorRT and GPU Optimization).

Step 3. "Identical accuracy" — false. Rounding error $∣ \overset{r}{^} - r ∣ \leq S /2$ per weight accumulates; outlier-sensitive layers can drop accuracy (Example 4). Fix: calibration or QAT. Why this step? connects to the parent's "quantizing is free accuracy" mistake.

Verify: only the size claim ( $10 MB$ ) is exact; the speed and accuracy claims are approximations. $40/4 = 10$ . ✓

Active recall

Recall Did every cell get covered?

Which example handles a value outside the calibrated range? ::: Example 4 (Cell D, saturation). Which example breaks the scale formula and why? ::: Example 5 — zero-width range gives $S = 0$ , dividing by zero in $Z$ . In the latency rule, edge wins when what exceeds what? ::: network+queue cost exceeds the compute penalty of the smaller chip. What makes per-channel ranges beat a global range mathematically? ::: smaller $S$ ⇒ smaller error bound $S /2$ (Example 6). When does infinite bandwidth flip the decision to cloud? ::: transfer terms vanish, so only queue+compute remain (Example 8).

Connections

Parent: Edge deployment and ONNX
Model Quantization and Pruning
Model Serving and Inference Latency
TensorRT and GPU Optimization
Computational Graphs
Data Privacy and Federated Learning
Mobile ML - CoreML and TFLite