Worked examples — Processing-in-memory (PIM)

The scenario matrix

Every PIM sizing question boils down to two quantities you must compute from a workload:

• Arithmetic intensity $I=C/N$ — compute ops per byte moved. Here $C$ is the number of arithmetic operations and $N$ is the number of bytes moved across the bus. Low $I$ = bandwidth-bound = PIM loves it. (See Roofline Model.)
• Bandwidth speedup ceiling $S_{\infty }=B_{int}/B$ — how much more internal bandwidth PIM unlocks vs the external bus. Here $B$ is the external bus bandwidth (bytes/second on the wire between memory and CPU), $k$ is the number of memory banks that can compute in parallel, and $B_{bank}$ is the bandwidth inside one bank. Their product $B_{int}=k{B}_{bank}$ is the total internal bandwidth PIM harvests.

The matrix below enumerates every case class this topic can throw at you. The seven examples afterward each cover one row (or the extreme of a row), and are labelled by cell.

Example 1 — Case A: streaming, very low intensity

1. Count compute ops. One add per element → $C=10^6$. Why this step? Intensity is ops-per-byte; we need the op count first.
2. Count bytes moved. Read b, read c, write a = 3 accesses × 4 bytes × $10^6$ = $12\times 10^6$ bytes. Why? The bus cost is set by every byte crossing it, reads and writes alike.
3. Compute intensity. $I=10^6/(12\times 10^6)=0.0833$ ops/byte. Why? This tiny number lands us far left of the red ridge → bandwidth-bound.
4. Internal bandwidth. $B_{int}=16\times 8=128$ GB/s. Why? Each bank computes in parallel; their internal bandwidths add (DRAM Organization (banks, rows)).
5. Speedup ceiling. $S_{\infty }=128/50=2.56\times$. Why not $16\times$? Because per-bank bandwidth ($8$) is below the bus ($50$); the win is the ratio of aggregate internal to external, not the bank count.

Verify: Units: $\frac{\text{ops}}{\text{byte}}$ ✓ dimensionless speedup ✓. Sanity: $I\approx 0.08\ll 1$ correctly flags a bandwidth-bound streaming kernel — exactly what PIM targets. $2.56\times$ is modest but real. Matches parent's Worked Example 2.

Example 2 — Case B: dense matmul, high intensity (PIM barely helps)

1. Ops. A dense matmul does $2n^3$ FLOPs (a multiply + an add per inner term): $C=2\times (1000{)}^3=2\times 10^9$. Why? $n^2$ output cells, each an $n$-length dot product = $n$ MACs = $2n$ flops.
2. Bytes (naïve, no reuse counted). Three $n\times n$ matrices: $N=3n^2\times 4=3\times 10^6\times 4=1.2\times 10^7$ bytes. Why? We move A, B, and C once each — the minimum traffic.
3. Intensity. $I=2\times 10^9/1.2\times 10^7\approx 166.7$ ops/byte. Why? This is far right of the red ridge → compute-bound.
4. Verdict. Each byte feeds ~167 ops → the CPU/GPU keeps its ALUs busy and caches hide the memory. PIM's data-movement win is nearly invisible here.

Verify: $I\approx 167\gg 1$ ✓ opposite extreme of Ex 1 ($0.083$). Ratio $167/0.083\approx 2000$ — the two workloads sit ~3.3 decades apart on the roofline, correctly on opposite sides of the ridge. See Neural Network Accelerators: they tile matmul precisely to raise reuse and stay compute-bound.

Example 3 — Case C: the break-even ridge

1. Write both time terms. Move time $=N/B$, compute time $=C/R$. Balance means they're equal. Why? The Roofline Model ridge is the exact crossover where the two costs match.
2. Set equal and solve. $N/B=C/R\Rightarrow C/N=R/B\Rightarrow I_{ridge}=R/B$. Why? Dividing through by $N$ turns raw counts into the intensity we can look up. Recall $R$ = peak compute rate (GFLOP/s), $B$ = bus bandwidth (GB/s).
3. Plug numbers. $I_{ridge}=100[\text{GFLOP/s}]/50[\text{GB/s}]=2$ ops/byte. Why? Below $2$ → bus-bound (PIM helps). Above $2$ → compute-bound (PIM meh).
4. PIM's effect. PIM raises the effective bandwidth from $B$ to $B_{int}$, which moves the ridge right to $I_{ridge}=R/B_{int}$. Workloads that were just barely compute-bound can fall back into the bandwidth-bound zone — but only ones near the ridge.

Verify: Units: $\frac{\text{GFLOP/s}}{\text{GB/s}}=\frac{\text{ops}}{\text{byte}}$ ✓. $I_{ridge}=2$. Ex 1 ($0.083<2$) is bandwidth-bound ✓; Ex 2 ($167>2$) is compute-bound ✓ — both classifications are consistent with this single ridge value.

Example 4 — Case D: degenerate, zero compute (pure copy)

1. Compute ops. A copy does no arithmetic: $C=0$. Why? This is the degenerate corner of the intensity axis.
2. Bytes. Read b, write a = 2 accesses × 4 bytes × $10^6$ = $8\times 10^6$ bytes. Why? Movement still happens — that's the whole point of the case.
3. Intensity. $I=0/(8\times 10^6)=0$ ops/byte — the extreme left edge of the roofline, below even Ex 1. Why? Zero over a nonzero number is a clean $0$; the ratio is well-defined (contrast: if $N$ were also $0$, it'd be $0/0$, undefined — that corner is drilled in Ex 8).
4. PIM verdict. Maximally bandwidth-bound. But note: PIM computes, and here there's nothing to compute. Near-memory copy (in-DRAM row-clone / RowClone) still helps by never crossing the bus; a compute-style crossbar does not.

Verify: $I=0$ ✓, the true floor of the intensity axis ($0<0.083<2<167$, ordering intact). Degenerate-input handled: $C=0$ gives a defined $I$; the $N=0$ corner is Ex 8.

Example 5 — Case E: limiting behaviour, infinite banks meets Amdahl

1. Bandwidth ceiling grows without bound. $S_{\infty }=k{B}_{bank}/B\to \infty$ as $k\to \infty$. Why? Nothing in the pure-bandwidth model caps it — that's the trap. Note $S_{\infty }$ is the ceiling of the parallel (bandwidth-bound) part only; it is exactly the "$S_{\text{parallel}}$" we feed into Amdahl next.
2. Bring in Amdahl's Law. Overall speedup $=1/(s+(1-s)/S_{\infty })$, where $s=0.05$ is the serial fraction and $S_{\infty }$ is the same bandwidth ceiling from step 1 — the only part Amdahl lets us accelerate. Why this tool? Amdahl answers exactly "what does the program speedup approach when one part goes infinitely fast?" — the right question here, not raw bandwidth.
3. Take the limit. As $S_{\infty }\to \infty$, $(1-s)/S_{\infty }\to 0$, so speedup $\to 1/s=1/0.05=20\times$. Why? The serial $5\%$ can't be sped up; it dominates once everything else is free.
4. Interpretation. Even with perfect infinite-bank PIM, this workload caps at $20\times$. The bus/serial residue is the wall.

Verify: $1/0.05=20$ ✓. Sanity: with a smaller serial fraction $s=0.01$ the cap would rise to $100\times$ — monotonic and correct. Limiting case handled: $k\to \infty$ gives a finite $20\times$, not infinity. Parent's warning "always sanity-check against Amdahl" ✓.

Example 6 — Case F: analog PUM crossbar accuracy

1. Ideal current per cell (Ohm's law). $I_{ij}=G_{ij}{V}_j$: $2\cdot 1=2$, $4\cdot 0.5=2$, $6\cdot 0=0$ (mA). Why Ohm's law? Each crosspoint is literally a resistor; $I=GV$ is the exact device physics.
2. Sum the row (Kirchhoff's current law). $I_i=2+2+0=4$ mA. Why? Currents merging on one wire add — that summation is the dot product ${\sum }_j{G}_{ij}{V}_j$.
3. Apply drift. First cell drifts $+3\%$: $G_1=2\cdot 1.03=2.06$ mS → its current $=2.06\cdot 1=2.06$ mA. New sum $=2.06+2+0=4.06$ mA. Why? Real conductances wander; this is why PUM is approximate, not exact.
4. Relative error. $(4.06-4)/4=0.015=1.5\%$. Why less than $3\%$? The drift hits one term contributing half the total, and the other terms are unaffected — errors dilute across the sum.

Verify: Ideal $=4$ mA, drifted $=4.06$ mA, error $=1.5\%$ ✓. Confirms parent's mistake-callout: crossbars are approximate, suited to error-tolerant nets, not exact finance.

Example 7 — Case G: word problem / exam twist (energy, not time)

1. Baseline energy. Move: $12\times 10^6\times 200\text{pJ}=2.4\times 10^9$ pJ. Compute: $10^6\times 1=10^6$ pJ. Total $E_{vN}=2.401\times 10^9$ pJ. Why split them? PIM only attacks the move term; separating shows how dominant it is.
2. Confirm movement dominates. $2.4\times 10^9$ vs $10^6$ → movement is $2400\times$ compute here. Why check? If compute dominated, saving movement would be pointless — this is the energy version of the intensity test.
3. PIM energy. Only $10\%$ of bytes move: $0.1\times 2.4\times 10^9=2.4\times 10^8$ pJ. Compute unchanged: $10^6$ pJ. Total $E_{PIM}=2.41\times 10^8$ pJ. Why compute unchanged? PIM doesn't make the arithmetic cheaper — it removes the trips.
4. Saving. $1-E_{PIM}/E_{vN}=1-2.41\times 10^8/2.401\times 10^9=0.8996\approx 90.0\%$. Why so large? Because movement was $2400\times$ compute, cutting $90\%$ of movement cuts ~$90\%$ of total energy.

Verify: $E_{vN}=2.401\times 10^9$ pJ, $E_{PIM}=2.41\times 10^8$ pJ, saving $\approx 90.0\%$ ✓. Units: pJ throughout ✓. Confirms the parent's core energy thesis: attack movement, not the ALU.

Example 8 — Case H: the fully degenerate corner, no data at all

1. Write the ratio. $I=C/N=0/0$. Why? This is the last unvisited corner of the intensity axis — both numerator and denominator vanish.
2. Classify the ratio. $0/0$ is indeterminate — mathematically undefined, not $0$ and not $\infty$. Why does this matter? Unlike Ex 4 (clean $0$ because only $C=0$), here you cannot place the workload anywhere on the roofline — there is no workload.
3. Physical reading. No bytes move → the bus was never the bottleneck; no ops run → there is nothing to accelerate. Both time terms $N/B$ and $C/R$ are $0$, so total time $\to 0$ regardless of architecture.
4. PIM verdict. PIM neither helps nor hurts: speedup is $0/0$ too. In practice a runtime guards against empty inputs before ever asking "should I PIM this?" — the intensity test is undefined and must be skipped.

Verify: $N=0\Rightarrow I=0/0$ undefined ✓ (contrast Ex 4's defined $I=0$). Both cost terms $N/B=0$ and $C/R=0$ ✓, so runtime is $0$ on any machine — the correct limiting reading of an empty kernel.

Active Recall