High Bandwidth Memory (HBM - HBM2 - HBM3) — Hardware notes

WHY does HBM exist?

Bandwidth for any memory interface is fundamentally:

WHY does "wide + slow" win on power? Dynamic power in a wire scales roughly as $P \propto C V^2 f$ . Short on-package wires have tiny capacitance $C$ and can use a small voltage swing $V$ , and low $f$ . So HBM moves a lot of data per joule.

WHAT is HBM physically?

Figure — High Bandwidth Memory (HBM - HBM2 - HBM3)

HOW to compute a stack's bandwidth (derivation)

Take HBM2:

Bus width per stack $N = 1024$ bits.
Per-pin data rate $f = 2.0\ \text{Gbps}$ (this is the effective rate; the physical clock is 1 GHz DDR = double data rate).

$\text{BW}_{\text{stack}} = \frac{N \times f}{8} = \frac{1024 \times 2.0\times10^{9}}{8}\ \text{B/s} = 256\ \text{GB/s}$

Why this step? $1024 \times 2.0\text{G} = 2.048\text{ Tbit/s}$ ; divide by 8 → 256 GB/s per stack. A 4-stack GPU → ~1 TB/s. That is the headline number.

Formula Generation cheat-sheet (per stack, 1024-bit bus)

Gen	Per-pin rate	Bus width	BW/stack	Typical max cap/stack
HBM	~1.0 Gbps	1024-bit	~128 GB/s	4 GB (4-Hi)
HBM2	~2.0 Gbps	1024-bit	~256 GB/s	8 GB (8-Hi)
HBM2E	~3.2–3.6 Gbps	1024-bit	~410–460 GB/s	16 GB
HBM3	~6.4 Gbps	1024-bit*	~819 GB/s	24 GB (12-Hi)
HBM3E	~9.6 Gbps	1024-bit	~1.2 TB/s	36 GB
*HBM3 doubles channels to 16 × 64-bit = still 1024 data bits total.

The physics never changed — every generation mostly raised per-pin $f$ and stacked more dies (higher "N-Hi").

Worked examples

Common mistakes (steel-manned)

Active recall

Recall Test yourself (hide answers)

What two knobs set bandwidth, and which does HBM push? → $N$ (width) and $f$ (rate); HBM pushes width.
What connects dies vertically inside a stack? → TSVs.
Why 2.5D not 3D-on-GPU? → Thermal: GPU heat would damage stacked DRAM.
HBM2 per-stack bandwidth? → 256 GB/s (1024 × 2 Gbps / 8).
Why is HBM energy-efficient? → Short, low-swing, low-frequency wires → tiny $CV^2f$ .

What is the fundamental bandwidth formula for a memory interface?

BW = N × f / 8, where N = number of data wires, f = per-pin transfer rate (bytes = bits/8).

Which parameter does HBM maximize to get bandwidth — width or per-pin speed?

Width (N ≈ 1024 bits/stack); per-pin speed is deliberately kept low.

What are TSVs?

Through-Silicon Vias — vertical copper interconnects drilled through a die to connect stacked DRAM dies.

What is a silicon interposer and what integration is it called?

A passive silicon carrier holding thousands of fine wires between stack(s) and processor; enables 2.5D integration.

Why is HBM 2.5D rather than true 3D on the GPU?

The GPU's heat would damage the stacked DRAM; side-by-side placement solves thermal + process-mismatch issues.

Compute HBM2 per-stack bandwidth.

1024 bits × 2.0 Gbps / 8 = 256 GB/s.

How does HBM3 keep a 1024-bit bus while doubling channels?

16 channels × 64-bit = 1024 data bits (vs HBM2's 8 × 128-bit).

Why is HBM more energy-efficient per bit than GDDR?

Short on-package wires → low capacitance C, small voltage swing V, low frequency f, so P ∝ CV²f is small.

What does "8-Hi" mean in HBM?

A stack of 8 DRAM dies (Hi = height/number of stacked dies).

Approx bandwidth of an HBM3E 4-stack system at 9.6 Gbps?

~1.23 TB/s/stack × 4 ≈ 4.9 TB/s.

Recall Feynman: explain to a 12-year-old

Imagine you must move a mountain of water. GDDR is like one super-fast fire hose — powerful but it whips around, gets hot, and wastes water. HBM is like opening a thousand garden taps side by side, right next to the bucket. Each tap is gentle, but together they fill the bucket way faster and don't overheat. HBM stacks the "taps" (memory chips) in a tower and sits them next to the brain (the chip) on a tiny glass tray so the water barely has to travel.

Connections

GDDR Memory — the "narrow + fast" alternative HBM competes with.
Memory Wall — the root problem HBM addresses.
Through-Silicon Vias (TSV) — the vertical interconnect technology.
2.5D and 3D Integration — packaging paradigms.
Silicon Interposer — the carrier enabling thousands of wires.
GPU Architecture — primary HBM consumer (AI accelerators, HPC).
Energy per Bit / pJ per bit — the efficiency metric HBM optimizes.
DDR vs GDDR vs HBM — comparison of memory families.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, problem yeh hai ki modern GPU aur AI accelerators ka compute bahut tez badh gaya hai, par memory se data laane ki speed (bandwidth) peeche reh gayi — isko "memory wall" bolte hain. HBM isko solve karta hai ek simple insight se: bandwidth ka formula hai BW = N × f / 8, jahan N = kitne wires aur f = har wire ki speed. GDDR wala approach kehta hai "kam wire, bahut tez chalao", jisse power aur heat dono badh jaate hain. HBM ulta karta hai — "bahut saare wire (1024 bits per stack), par har wire slow aur low-voltage" chalao. Result: same ya zyada bandwidth, magar bahut kam power.

Physically HBM ka trick yeh hai ki DRAM chips ko ek doosre ke upar stack kar dete hain, aur unko vertically jodte hain TSVs (Through-Silicon Vias) se — matlab silicon ke through drilled copper pillars. Yeh poora stack GPU ke bagal mein ek silicon interposer par baithta hai, jise 2.5D integration kehte hain. GPU ke upar seedha nahi rakhte kyunki GPU garam hota hai aur DRAM jal jaayega — isliye side-by-side.

Har generation mein zyada kuch nahi badalta: bas per-pin speed f badhti jaati hai aur zyada dies stack hote hain (8-Hi, 12-Hi). HBM2 = 256 GB/s per stack, HBM3 = ~819 GB/s per stack, HBM3E ~1.2 TB/s. Chaar stacks laga do to poore accelerator ki bandwidth terabytes/second mein pahunch jaati hai. Yaad rakhne ka mantra: Wide, Short, Stacked — chaudi bus, chhote wire, upar stack kiye hue chips. Yehi HBM ki poori kahani hai.

High Bandwidth Memory (HBM - HBM2 - HBM3)

WHY does HBM exist?

WHAT is HBM physically?

HOW to compute a stack's bandwidth (derivation)

Worked examples

Common mistakes (steel-manned)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Advanced & Emerging Architectures

Connections