Superscalar — multiple execution units: Visual walkthrough

We will slowly assemble one central result:

$IPC_{eff} = \frac{1}{( 1 - p ) + \frac{p}{N}}$

Don't worry — right now that is just a shape. By Step 8 every letter in it will be a thing you can point at in a drawing.

Step 1 — What a "clock cycle" even is

WHAT we do first: agree that time inside a CPU is not smooth — it comes in equal steps, like frames of a film.

WHY we start here: our whole story is about how many instructions finish per tick. If you don't picture the ticks, you can't count anything.

PICTURE: look at the figure. The bottom is a ruler of identical boxes; each box is one tick. The red marker shows a single instruction finishing at the end of tick 3.

Figure — Superscalar — multiple execution units

Recall

What is $T_{cycle}$ ? ::: The length in seconds of one clock cycle (one CPU tick).

Step 2 — An "instruction" and a single-lane pipeline

A pipeline is an assembly line: each instruction walks through stages (fetch → decode → execute → write result). Like a car wash, a new instruction can enter while an earlier one is still inside — so once the line is full, one instruction pops out the end each tick.

WHAT: we set the baseline machine — a scalar pipeline, one lane.

WHY: to measure a speedup you need a "before". The before is: at best, 1 instruction per tick. That number is our starting ceiling.

PICTURE: the single red lane. Instructions $I_{1}, I_{2}, I_{3}$ file through in a line; exactly one exits per tick. Nothing overtakes.

Step 3 — Naming the score: IPC

WHAT: we turn "how good is this machine" into one number.

WHY tan-style — why this number and not raw seconds? Seconds mix three separate things together: how many instructions, how many ticks each needs, and how long a tick is. IPC isolates the middle thing — the one superscalar actually changes. That is exactly the quantity we want to watch, so we give it its own name.

PICTURE: two scoreboards. Left: 3 instructions over 3 ticks → IPC $= 1$ . Right (a peek ahead): 6 instructions over 3 ticks → IPC $= 2$ .

Step 4 — Add lanes: the superscalar idea

WHAT: we widen the single lane of Step 2 into $N$ parallel lanes.

WHY: transistors are cheap, time is precious. If two adders exist, two independent additions can happen in the same tick — the finished-count per tick can now reach $N$ instead of $1$ .

PICTURE: $N = 2$ . Two red lanes running in parallel; in tick 1 both $I_{1}$ and $I_{2}$ pop out together. The scoreboard reads IPC $= 2$ .

Step 5 — The catch: dependencies

WHAT: we discover why the extra lanes sometimes sit empty.

WHY: hardware can run two adds at once only if both have their inputs ready. If lane 2's instruction is waiting for lane 1's answer, lane 2 must idle. Extra lanes can't invent data that doesn't exist yet.

PICTURE: two lanes, but $I_{2}$ needs $I_{1}$ 's result (red arrow from $I_{1}$ to $I_{2}$ ). Lane 2 stalls in tick 1 (greyed, empty) and only runs in tick 2. The chain forces the machine back to one finish per tick.

Step 6 — Splitting a program into "parallel" and "serial" parts

WHAT: to predict real speed, sort a program's instructions into two buckets.

Parallel bucket — a fraction $p$ of instructions that are mutually independent, so they can spread across the $N$ lanes freely.
Serial bucket — the remaining fraction $(1 - p)$ that forms dependency chains, so they must go one-after-another, no matter how many lanes you own.

WHY: these two buckets behave completely differently under widening. Parallel work gets divided by $N$ ; serial work is immune to $N$ . To predict the total we must treat them separately and then add. ( $p$ is just a fraction between $0$ and $1$ : $p = 1$ means "everything is independent", $p = 0$ means "everything is one long chain".)

PICTURE: a bar of total work. A red portion of length $p$ is labelled "parallel"; the black portion $(1 - p)$ is labelled "serial chain".

Step 7 — Building the effective-IPC formula, term by term

Now we assemble the central result. Think in time to do one instruction's worth of work (that is CPI-flavoured), then flip to IPC at the end.

Serial part. A fraction $(1 - p)$ of the work runs one-at-a-time. One-at-a-time means 1 cycle of "width-time" per instruction — lanes don't help. Its time contribution is: $serial share (1 - p) \times no speedup: N doesn’t help 1$

Parallel part. A fraction $p$ of the work spreads across $N$ lanes, so it finishes in $1/ N$ of the width-time it would otherwise need: $parallel share p \times split over N lanes \frac{1}{N} = \frac{p}{N}$

Add them — total width-time per instruction (this is effective CPI): $CPI_{eff} = (1 - p) + \frac{p}{N}$ Each piece: $(1 - p)$ is the stubborn chain that ignores your lanes; $\frac{p}{N}$ is the friendly work you actually got to divide.

Flip to IPC (Step 3: IPC $= 1/$ CPI):

PICTURE: two stacked time-bars — a scalar machine ( $N = 1$ ) vs a wide machine. Only the red (parallel) chunk shrinks; the black (serial) chunk is exactly the same length in both. That unshrinkable black chunk is $(1 - p)$ , and it is what caps your speed.

Step 8 — Reading every case off the formula

WHAT: plug in the extreme and degenerate values so no scenario surprises you.

Case	$p$	$N$	Denominator $(1 - p) + \frac{p}{N}$	$IPC_{eff}$	Meaning
Fully parallel	$1$	$N$	$0 + \frac{1}{N} = \frac{1}{N}$	$N$	Hit the ceiling — every lane busy.
Fully serial chain	$0$	any	$1 + 0 = 1$	$1$	Width useless (Step 5's chain).
Half & half	$0.5$	$2$	$0.5 + 0.25 = 0.75$	$1.333$	Typical: below ceiling.
Half & half, infinite lanes	$0.5$	$\infty$	$0.5 + 0 = 0.5$	$2$	Even $\infty$ lanes stall at $\frac{1}{1 - p}$ .
No lanes at all	any	$1$	$(1 - p) + p = 1$	$1$	Scalar baseline — formula reduces correctly.

WHY these matter: the last row is the sanity check — set $N = 1$ and the formula must give the scalar machine of Step 2. It does. The $N = \infty$ row is the sobering one: no amount of hardware beats $\frac{1}{1 - p}$ .

PICTURE: a curve of $IPC_{eff}$ against $N$ for a fixed $p = 0.5$ . It rises then flattens into a red horizontal ceiling at $\frac{1}{1 - p} = 2$ . Adding lanes past a point buys almost nothing.

Recall

With $p = 0.5$ , what does IPC approach as $N \to \infty$ ? ::: $\frac{1}{1 - p} = \frac{1}{0.5} = 2$ . With $p = 0$ , what is IPC for any $N$ ? ::: Exactly $1$ — a pure chain ignores width.

The one-picture summary

This single figure compresses the whole walkthrough: on the left, one lane (scalar, IPC $= 1$ ); in the middle, $N$ lanes with a red dependency arrow forcing an idle lane; on the right, the IPC-vs- $N$ curve bending into its red ceiling at $\frac{1}{1 - p}$ . Follow the arrows left-to-right and you have re-derived the central result.

Recall Feynman retelling — the whole page in plain words

Time in a CPU comes in equal ticks (Step 1). A normal CPU is a single-lane car wash: one instruction finishes per tick, so its "score" — instructions per tick, called IPC — is 1 (Steps 2–3). A superscalar CPU bolts on extra lanes; with $N$ lanes it could finish $N$ instructions per tick (Step 4). But instructions often need each other's answers first — a dependency — and a lane waiting for data just sits empty (Step 5). So we split a program into the part that's independent (fraction $p$ , spreads over the lanes) and the part that's a chain (fraction $1 - p$ , stuck going one at a time). Adding the time for both and flipping to IPC gives $IPC_{eff} = 1/ ((1 - p) + p / N)$ (Steps 6–7). Plug in the extremes and you see the truth: fully independent → you hit $N$ ; a pure chain → you're stuck at 1 forever; and even infinitely many lanes can't beat $1/ (1 - p)$ (Step 8). More cooks help only when there are independent sandwiches to make.

Real machines fight the chain with Out-of-Order Execution, Register Renaming, the Reorder Buffer (ROB), and Branch Prediction — but they can never delete the $(1 - p)$ that this one page just made you draw.