Visual walkthrough — Simultaneous multithreading (SMT - hyperthreading)

The parent note claims that two-way SMT gives a throughput speedup of about $1.6\times$ — not the $2\times$ you might expect. This page builds that number from absolute zero, one picture at a time. We assume nothing except: a processor does work each clock tick, and sometimes it can't.

If you have not yet met the idea of many execution units working at once, peek at Superscalar Processors first — but everything you need is re-built here.

---

Step 1 — What a "cycle" and an "execution unit" actually are

WHAT. A processor runs to the beat of a clock. Each tick is one cycle. Inside the core there are little worker-machines called execution units — an ALU adds numbers, an FPU handles decimals, a load/store unit talks to memory. We write the number of these units as $N$; in our running example $N=4$, so four pieces of work can happen in one tick.

WHY start here. Every quantity later — utilization, IPC, speedup — is just counting boxes filled per tick. If you can see the boxes, the algebra is bookkeeping.

PICTURE. Look at the grid below. Each column is one cycle (one tick of the clock). Each row is one execution unit — so there are $N=4$ rows. A filled box means "this unit did useful work this cycle"; an empty box means "idle — nobody had work ready for it."

---

Step 2 — Why a single thread leaves boxes empty

WHAT. A thread is one running program: a stream of instructions with a Program Counter (PC) marking "where am I." Feed one thread into our $N=4$-wide core and count the filled boxes.

WHY. Programs are not smooth. Three things create empty columns (the parent calls them bubbles):

1. Data dependencies — instruction $I_2$ needs $I_1$'s answer, so it must wait (see Pipeline Hazards).
2. Cache misses — memory is far away; a miss freezes the thread for 50–200 cycles (see Memory Hierarchy).
3. Branch mispredictions — a wrong guess flushes work (see Branch Prediction).

PICTURE. The red block below is a cache-miss stall: every unit is idle because the thread has no ready instruction to give anyone. The hardware exists; it simply has no work.

$$\underset{\text{fraction of boxes filled}}{\underbrace{{\text{Utilization}}_{\text{single}}}}=\frac{\stackrel{\text{filled boxes / column}}{\overbrace{\text{IPC}}}}{\underset{\text{rows per column}}{\underbrace{N}}}\approx 20\text{–}40\%$$

Read it as: filled boxes ÷ all $N$ boxes in a column. Most of the grid is empty. That waste is the whole opportunity.

---

Step 3 — Turn IPC into a time (the baseline number)

WHAT. We want a concrete cycles figure to compare against. Time is just work divided by rate. Fix a workload of $W=1000$ instructions per thread — we will reuse this same $W$ for every case on this page so the comparisons stay honest.

WHY division. IPC is "instructions per cycle." Flip it: cycles per instruction is $1/\text{IPC}$. Multiply by how many instructions we owe and the cycles fall out. This is the only tool we need — no calculus, just the definition of a rate.

$$T_{\text{single}}=\frac{\stackrel{=1000\text{instructions}}{\overbrace{W}}}{\underset{=1.5\text{work per cycle}}{\underbrace{\text{IPC}}}}=\frac{1000}{1.5}=667\text{cycles}$$

With IPC $=1.5$ on an $N=4$-wide core, utilization is $\frac{1.5}{4}=37.5\%$ — five-eighths of the grid sits empty.

PICTURE. The bar shows 667 cycles of wall-clock time, colour-split into the ~37.5% that did work and the ~62.5% wasted on stalls.

---

Step 4 — Drop a second thread into the empty boxes

WHAT. Now replicate the cheap state — a second PC and a second architectural register file — so the core holds two thread contexts at once. Everything expensive (the four execution units, the caches, the branch predictor) stays shared.

WHY it works. When thread A stalls (its column would be empty), thread B usually has unrelated work ready — B's cache miss and A's cache miss don't happen at the same instant. So B's filled boxes slot into A's empty ones.

PICTURE. Same grid as Step 1, now two-coloured: blue boxes = thread A, pink boxes = thread B. Notice the red stall column from Step 2 is now mostly pink — B filled A's hole.

---

Step 5 — Count the combined work, and why each thread slows a little

WHAT. With both threads live, measure IPC again. Each thread now runs at IPC $=1.2$, lower than its solo $1.5$.

WHY the slowdown? They now share finite things:

• cache capacity → more misses each (see Cache Coherence for what happens when they touch the same data),
• reorder-buffer slots → a smaller out-of-order window each,
• one branch predictor → they overwrite each other's history.

But the combined rate rises (still capped by $N=4$):

$${\text{IPC}}_{\text{combined}}=\underset{\text{thread A}}{\underbrace{1.2}}+\underset{\text{thread B}}{\underbrace{1.2}}=2.4\le N$$

Utilization climbs to $\frac{2.4}{4}=60\%$. Per thread the time is worse (same $W=1000$):

$$T_{\text{SMT}}=\frac{W}{\underset{\text{slower per-thread IPC}}{\underbrace{1.2}}}=\frac{1000}{1.2}=833\text{cycles}$$

...but both threads finish in that same 833 cycles, because they ran together, not one-after-the-other.

PICTURE. Two stacked bars. Top: solo thread, 667 cycles, mostly empty. Bottom: SMT, 833 cycles but doing two threads' worth of work — visibly fuller.

---

Step 6 — The speedup number

WHAT. Speedup = (work rate with SMT) ÷ (work rate without).

WHY a ratio. "Speedup" only means anything as a comparison. Put total instructions over total cycles for each case, then divide.

$$\text{Speedup}=\frac{\stackrel{\text{SMT: 2 threads’ work / shared time}}{\overbrace{2W/833}}}{\underset{\text{solo: 1 thread’s work / its time}}{\underbrace{W/667}}}=\frac{2.4}{1.5}=1.6\times$$

Both numerator and denominator are just "instructions ÷ cycles," i.e. IPC — the $W$'s cancel — so the whole thing collapses to $\frac{2.4}{1.5}$. The $2\times$ dream fails by exactly the amount the threads slowed each other (Step 5).

---

Step 7 — The two extreme cases (why "it depends")

WHAT. Speedup swings with what the threads do. We show both ends, keeping the same $W=1000$ instructions per thread as everywhere else — only the IPC changes, so every cycle count below is just $1000/\text{IPC}$.

Case A — complementary (memory-bound + CPU-bound). Thread A is pure compute, solo IPC $=2.0$; thread B is a pointer-chase that stalls most cycles, solo IPC $=0.5$.

• Solo cycles: A $=\frac{1000}{2.0}=500$, B $=\frac{1000}{0.5}=2000$.
• Sequential (run one, then the other): $500+2000=2500$ cycles.
• SMT: A's compute drops into B's stall-holes, so A barely slows (IPC $\to 1.8$) and B slightly speeds (IPC $\to 0.6$): A $=\frac{1000}{1.8}\approx 556$, B $=\frac{1000}{0.6}\approx 1667$.
• Both run together, so finish $=\max (556,1667)=1667$.
• Speedup $=2500/1667\approx 1.5\times$. Big win — the long memory pole hid A's work inside it.

Case B — identical (both CPU-bound). Both threads hammer the ALUs, solo IPC $=2.5$ each.

• Solo cycles: each $=\frac{1000}{2.5}=400$.
• Sequential: $400+400=800$ cycles.
• SMT: no empty boxes to steal, so contention splits the units and each drops to IPC $=1.4$: each $=\frac{1000}{1.4}\approx 714$.
• Both together, so finish $=714$ cycles.
• Speedup $=800/714\approx 1.12\times$. Almost nothing.

WHY show both. SMT is opportunistic, not magic: it can only fill boxes that were empty. Two greedy threads leave no gaps.

PICTURE. Left panel: complementary threads interlocking like puzzle pieces (near-full grid). Right panel: identical threads colliding, grid already full so no gain.

---

Step 8 — Degenerate limits and the model behind them

WHAT. Cases A and B suggest the answer depends on how often the units sit idle. Let $f$ be the fraction of a thread's cycles that are memory-bound (execution units idle, boxes empty) — so $(1-f)$ is the fraction that is CPU-bound (units already full). We now build a formula for the SMT speedup directly from these two fractions.

WHY split the timeline this way. SMT can only help where there is an empty box. So split a thread's total time into the two kinds of cycle and handle each on its own:

• CPU-bound cycles, fraction $(1-f)$. The execution units are already saturated — the grid column is full. A second thread finds no empty box, so these cycles get no help: they still cost their full share. This contributes the term $(1-f)$ to the SMT running time.
• Memory-bound cycles, fraction $f$. The column is empty, so the second thread can fill boxes. Conservatively assume it fills half of the wasted work — the two threads share the idle window. So this chunk now costs only $\frac{f}{2}$ instead of $f$. This contributes the term $\frac{f}{2}$.

Add the two chunks to get the SMT running time (as a fraction of the solo time), then invert to turn "less time" into "more speed":

$${\text{time}}_{\text{SMT}}=\underset{\text{full cycles, no help}}{\underbrace{(1-f)}}+\underset{\text{idle cycles, 2nd thread fills half}}{\underbrace{\frac{f}{2}}}⟹\text{Speedup}=\frac{1}{(1-f)+\frac{f}{2}}$$

The $1$ on top is the solo running time (100% of itself); dividing gives how many times faster SMT is.

PICTURE. The speedup curve from $f=0$ to $f=1$: flat near $1$ on the left (nothing to fill), rising to $2$ on the right (everything can be filled), with our three checkpoints marked.

Now push $f$ to its edges as a sanity check:

$f$ (memory-bound share)	meaning	speedup
$f=0$	never stalls, grid always full	$\frac{1}{1+0}=1.0\times$ (no gain)
$f=1$	always stalling, grid always empty	$\frac{1}{0+0.5}=2.0\times$ (ideal ceiling)
$f=0.6$	heavily memory-bound	$\frac{1}{0.4+0.3}\approx 1.43\times$

Why these matter. They bracket reality. You can never do worse than $1\times$ (SMT never removes work already fitting) nor better than $2\times$ with two threads (you only have two threads to fill boxes). Every real workload lands strictly between.

---

The one-picture summary

Everything above in a single frame: the empty solo grid, the second thread pouring into the holes, and the resulting $1.6\times$ arrow — plus the reminder that the gain lives only in the empty boxes.

Recall Feynman retelling — say it in plain words

A processor is a factory row of four machines, ticking once per clock. One worker (a thread) hardly ever keeps all four busy — half the time he's waiting for parts to arrive from the far-away warehouse (memory), and every machine just sits there. So we invite a second worker onto the same floor. He doesn't need his own machines — he uses the ones standing idle whenever the first worker is stuck waiting. Now, in the same amount of time, roughly one-and-a-half to two times as many parts get built. But two catches: they share the warehouse, so both wait a bit longer for parts (each is a touch slower alone); and if both workers only ever want the same machine, there are no idle machines to share and you gain almost nothing. So SMT is worth a lot when the two jobs need different things, and almost nothing when they need the same thing.

Recall

What single quantity, read off the grid, is "filled boxes per column"? ::: IPC (instructions per cycle) — average execution units doing useful work each cycle, capped at $N$. Why is two-way SMT speedup $1.6\times$ and not $2\times$ in the worked example? ::: Because shared caches, ROB slots and the branch predictor drop each thread from IPC $1.5$ to $1.2$; combined $2.4/1.5=1.6$. In the model, what speedup do you get at $f=0$ and $f=1$? ::: $1.0\times$ (never stalls, no empty boxes to fill) and $2.0\times$ (always stalls, second thread fills half of everything). When does SMT help most? ::: When threads are complementary — one memory-bound (leaves holes), one CPU-bound (fills them).