Processing-in-memory (PIM): Visual walkthrough — Hardware deep dive

Everything below is built from zero. If a symbol appears, it was drawn first.

Step 1 — What actually happens when a program touches data

WHY split them? Because if we lump "running a program" into one blob of time, we can't see which part is slowing us down. The whole PIM argument rests on the fact that movement and compute are different, and that one of them dominates. So the first move is to name them apart.

Let me define the two quantities we will track for the rest of the page:

$N$ = the number of bytes the workload must move (a byte = 8 bits, the unit memory is measured in).
$C$ = the number of compute operations (an "op" = one arithmetic action, like one add).

PICTURE. The chef-and-warehouse cartoon, made literal. The number lives far away in memory; a long wire carries it to the processor; only there does the "+" happen.

Look at the two arrows: the long red one is movement, the short green one is compute. Step 2 asks which one costs more.

Step 2 — Movement is the expensive part (the energy picture)

WHY does this decide everything? If the expensive part is compute, you speed things up by building a faster arithmetic unit. If the expensive part is movement, a faster arithmetic unit changes nothing — you'd just wait at a faster ALU. So we must find out which term dominates before we design anything.

To compare fairly, both energies must be measured per operand of the same size — say one 4-byte (32-bit) word, the natural unit an "add" consumes. That way $E_{m o v e}$ and $E_{o p}$ refer to the exact same chunk of data, and their ratio is meaningful.

PICTURE. Two bars on the same axis: a tiny green bar (compute energy) beside a towering red bar (move energy), both for one word. The height gap is the memory wall in energy form.

Step 3 — Writing down the total time on a classic machine

WHY this shape? Time = work / rate is the most basic timing law there is: if you move $N$ bytes at $B$ bytes per second, it takes $N / B$ seconds. Same logic for compute. We use it because it isolates the bottleneck cleanly — each fraction is one independent stage.

Two new rates enter, each defined before use:

$B$ = bandwidth: how many bytes per second the external bus (the long wire) can carry.
$R$ = compute rate: how many operations per second the processor can execute.

PICTURE. A horizontal timeline split into two colored segments: a long red segment ( $N / B$ ) followed by a short green one ( $C / R$ ). The length of each segment is literally its formula.

Step 4 — Arithmetic intensity: which segment wins?

WHY invent this ratio? Because $T_{v N}$ has two terms and we want to know which one is bigger without plugging in machine speeds. Dividing compute-work by move-work strips the timeline down to one comparable number. This is exactly the axis of the Roofline Model.

For a data-hungry job (streaming, low reuse), $I$ is tiny, so $C / R$ is negligible next to $N / B$ :

$T_{v N} \approx \frac{N}{B} (bandwidth-bound)$

PICTURE. Two timelines stacked: top one has $I$ small (mostly red), bottom has $I$ large (mostly green). The dividing question — "which color fills the bar?" — is the PIM go/no-go test.

Step 5 — Where PIM attacks: the hidden internal bandwidth

WHY does this matter? The bottleneck $B$ was the external straw. If we compute inside memory, we never squeeze through that straw — we use the wide internal read instead. So we must name the internal bandwidth.

$k$ = number of banks that can work in parallel.
$B_{bank}$ = bandwidth of one bank's internal read.
$B_{in t}$ = the combined internal bandwidth.

PICTURE. A wide reservoir (many parallel bank-pipes, each $B_{bank}$ ) all feeding one narrow external straw $B$ . PIM lets you drink from the reservoir directly.

Step 6 — The PIM timeline and the speedup formula

WHY simply swap $B \to B_{in t}$ ? Because the only thing PIM changes is the rate at which operands reach the arithmetic unit. Compute work $C$ and compute rate $R$ are unchanged. Movement work $N$ is unchanged. Only the road got wider.

PICTURE. Two timelines drawn to scale, one on top of the other: the von Neumann bar (long red) above the PIM bar (short red). The ratio of the red lengths is the speedup.

Step 7 — The edge cases (where the formula lies)

WHY do these failure modes even exist? Look back at the two things the derivation assumed. Step 6's cancellation of $N$ only worked because we dropped the compute term $C / R$ (assumption 1: bandwidth-bound). And the speedup counts a whole program as if it were one bandwidth-bound kernel (assumption 2: everything benefits). Each case below is exactly what happens when one of those hidden assumptions breaks — so each one modifies a specific line of the derivation, not the picture in general.

Case A — Compute-bound ( $I$ large). Which assumption breaks: assumption 1. If $C / R$ is not negligible, the $N$ terms no longer dominate, so the cancellation in Step 6 never happens. Then $T_{P I M} \to C / R$ , the same as von Neumann, and speedup $\to 1$ . Widening the road doesn't help if you were never stuck on the road. This is the roofline ceiling.

Case B — Amdahl's ceiling. Which assumption breaks: assumption 2. Only the movement-bound fraction of the program is sped up; the rest runs at its old speed. If a fraction $f$ of the work is bandwidth-bound and gets sped up by $s = B_{in t} / B$ , then Amdahl's Law replaces the raw speedup $s$ with $Speedup_{t o t a l} = \frac{1}{( 1 - f ) + f / s},$ which even at $s = \infty$ cannot beat $1/ (1 - f)$ . The un-accelerated $(1 - f)$ portion sets the ceiling.

Case C — Degenerate / analog input. Which assumption breaks: the "op is exact" premise hidden inside $C / R$ . For crossbar-style compute using the memory's own physics, the "op" is not digital — it is an analog current summed on a wire. Define fresh symbols (they are electrical, not the intensity $I$ from Step 4):

$G$ = conductance of a memory cell (how easily current flows, in siemens).

$V$ = input voltage applied to a column (in volts).

$I_{ce l l} = G V$ = current through that cell (Ohm's law); currents on a shared row wire add (Kirchhoff), giving the dot product.

Zero input voltage ( $V = 0$ ) → zero current (a harmless degenerate case), but conductance drift and analog-to-digital noise make the summed result approximate. The speedup is real; the exactness is not. Use it for error-tolerant neural nets, never for exact finance.

PICTURE. Three mini-panels: (A) a timeline still green-dominated after PIM (no win); (B) an Amdahl curve flattening to $1/ (1 - f)$ ; (C) an ideal straight line $I_{ce l l} = G V$ with a fuzzy noisy band around it.

The one-picture summary

Everything in one frame: the long von Neumann timeline shrinks because the fat external-bus segment is replaced by a thin internal-bandwidth segment, while the compute segment (unchanged) is what eventually limits you.

Recall Feynman retelling — the whole walkthrough in plain words

A program does two jobs: fetching numbers and doing math on them. Fetching them across the long wire is slow and burns 100–1000× the energy of the math (Steps 1–2, both measured per 32-bit word). So we wrote total time as "move time + compute time" = $N / B + C / R$ (Step 3). To know which part hurts, we divide ops by bytes to get arithmetic intensity $I = C / N$ ; if it's small, we're stuck waiting on the wire (Step 4). PIM's trick: memory is already wide inside — many banks, each reading thousands of bits at once — so its internal bandwidth $B_{in t} = k B_{bank}$ dwarfs the skinny external bus $B$ (Step 5). Compute there and the move segment shrinks by exactly $B_{in t} / B$ ; that ratio is the speedup (Step 6). But it only helps if you were bandwidth-bound, it's capped by how much of your program is bandwidth-bound (Amdahl), and analog versions trade exactness for speed (Step 7). One sentence: don't move the data — move the math.