Average memory access time (AMAT): Visual walkthrough

We assume you know only this: memory is where the computer keeps data, and some memory is close/fast while some is far/slow. Everything else we define as we go.

Step 1 — What a single memory access even is

WHAT: We picture the CPU sending one request into a fork in the road — a hit path and a miss path.

WHY: Before averaging anything, we must know exactly the two things we are averaging over. Every access is one or the other — there is no third outcome.

PICTURE: The green door (hit) delivers data quickly; the red road (miss) is a long detour to the slow box.

Step 2 — Naming the two time costs

WHAT: We attach a stopwatch value to each path.

WHY this split, and not "total time on a miss"? Because on a miss you still had to check the cache first — you didn't skip that step. So the miss path is "cache check, then detour" = $H T + M P$ , not just $M P$ . Getting this bookkeeping right is the whole ballgame; see Miss Penalty & Main Memory Latency.

PICTURE: The green path is one bar of length $H T$ . The red path is that same bar $H T$ followed by a much longer bar $M P$ — the detour is glued on after, never instead.

Step 3 — How often does each path happen? (Enter miss rate)

WHAT: We label the fork with probabilities: red road taken a fraction $m$ of the time, green door a fraction $1 - m$ .

WHY fractions and not counts? Because "average time per access" shouldn't depend on how many accesses we ran — only on the proportions. A fraction captures "how likely is each path" independent of total volume. Why hit rates are high in real life is the job of Locality of Reference.

PICTURE: Out of $100$ little request-dots pouring into the fork, $1 - m$ of them stream through the green door and $m$ trickle down the red road.

Step 4 — What "average" means here (weighted, not plain)

WHAT: We recall the one tool we need — a weighted average — and say why it's the right tool.

WHY this tool and not a plain average? A plain average $(a + b) /2$ pretends both outcomes are equally common. They are not: hits are common, misses are rare. We must weight each cost by its frequency, or the rare-but-huge miss would be wildly over- or under-counted. The weighted average is exactly the machine that says "count each cost in proportion to how often you pay it."

PICTURE: Two stacks of coins — a tall stack (hits) each worth a little, a short stack (misses) each worth a lot. The average is where the see-saw balances, pulled toward the tall stack but tugged by the heavy few.

Step 5 — Assemble AMAT from the two paths

WHAT: Plug our two paths into the weighted-average machine.

Hit path: weight $(1 - m)$ , cost $H T$
Miss path: weight $m$ , cost $H T + M P$

AMAT = fraction of hits (1 - m) cost of a hit H T + fraction of misses m cost of a miss (H T + M P)

WHY written this way? It is the literal readout of Step 4's picture: each stopwatch value multiplied by how often that path is taken, then summed. Nothing invented — just the two forks, weighted.

PICTURE: The two coin-stacks from Step 4 relabeled with our actual times, feeding into one balance point marked "AMAT."

Step 6 — Simplify: watch the hit time collapse

WHAT: Multiply out and collect like terms.

AMAT = (1 - m) H T + m H T + m M P

Look at the first two terms — the pieces carrying $H T$ :

H T on the (1 - m) hits (1 - m) H T + H T on the m misses m H T = [(1 - m) + m] H T = every access! 1 \cdot H T = H T

The $- m H T$ and $+ m H T$ cancel. What survives:

AMAT = H T + m \cdot M P

WHY does the $H T$ lose its weighting? Because $H T$ is paid on both paths — hits and misses. Adding " $H T$ on the hits" plus " $H T$ on the misses" is just " $H T$ on all accesses" = a flat, unconditional $H T$ . Only the penalty $M P$ lives solely on the miss path, so only it keeps the weight $m$ .

PICTURE: The $H T$ bars from both paths slide together and merge into a single unavoidable block; only the red $M P$ bar keeps a fractional-height $m$ shadow.

Step 7 — Edge cases: does the formula survive the extremes?

WHAT: Push $m$ to its two boundaries and check the answer stays sane.

Case	Set	Result	Reads as
Never miss	$m = 0$	$AMAT = H T + 0 = H T$	Every access is instant-ish — you only ever pay the fast hit.
Always miss	$m = 1$	$AMAT = H T + M P$	Every access checks the cache and detours — the full miss path, every time.
Small $m$ , huge $M P$	e.g. $m = 0.01$ , $M P = 200$	$H T + 2$	Rare misses still add real time — that's why we care.

WHY test these? A formula you can't stress-test is a formula you don't understand. At $m = 0$ it must reduce to pure hit time (no misses = no penalty). At $m = 1$ it must equal the full miss cost $H T + M P$ (Step 2's red bar). Both hold — the formula is trustworthy across the entire range $0 \leq m \leq 1$ .

PICTURE: A slider for $m$ from $0$ to $1$ ; AMAT rises as a straight line from $H T$ (at left) to $H T + M P$ (at right). Its slope is exactly $M P$ .

Step 8 — Nesting: the miss penalty is itself an AMAT

WHAT: Replace the single slow box with two boxes — an L2 cache in front of DRAM (see Multi-level Cache Hierarchy). Now an L1 miss doesn't go straight to DRAM; it first asks L2.

So the L1 miss penalty is no longer a fixed number — it's "the average time to satisfy the request starting at L2," which is itself an AMAT:

M P_{L 1} = always check L2 H T_{L 2} + fraction of L2 misses m_{L 2} \cdot DRAM detour M P_{L 2}

Substitute into Step 6's formula:

AMAT = H T_{L 1} + m_{L 1} (H T_{L 2} + m_{L 2} \cdot M P_{L 2})

WHY does this crush the average? Multiply out the DRAM term: it arrives as $m_{L 1} \cdot m_{L 2} \cdot M P_{L 2}$ . Two small fractions multiplied make a tiny fraction — so the terrifying DRAM cost is paid on almost no accesses. That product $m_{L 1} \cdot m_{L 2}$ is the global miss rate of L2. Inside the formula we use $m_{L 2}$ local (only accesses that reached L2), because L2 only ever sees L1's leftovers.

PICTURE: A Russian-doll diagram: the red "miss" road of L1 opens into a second smaller fork (L2 hit vs DRAM detour) — the same picture from Step 1, nested inside itself.

The one-picture summary

Everything above, on one canvas: the fork (Step 1), the two glued time-bars (Step 2), the fractions (Step 3), the weighted balance (Step 4–5), the collapse to $H T + m \cdot M P$ (Step 6), the straight-line dependence on $m$ (Step 7), and the nested L2 fork (Step 8).

Recall Feynman retelling — the whole walkthrough in plain words

Every time the computer wants data, it knocks on the fast cache door. Most of the time the data is right there — quick! That's a hit, and knocking took time $H T$ . Once in a while the data isn't there — a miss — so after knocking it has to run all the way to the slow warehouse and back, an extra cost $M P$ . To find the average wait, we don't treat hits and misses as equally common, because they aren't: we weight each by how often it happens ( $1 - m$ hits, $m$ misses). When we add it all up, the knock ( $H T$ ) got paid on every trip — hit or miss — so it stands alone as a flat cost. Only the warehouse run ( $M P$ ) happens on the rare misses, so it gets shrunk by the small fraction $m$ . That's the whole formula: $AMAT = H T + m \cdot M P$ . Test it: if you never miss, you just pay the knock; if you always miss, you pay knock-plus-run every time — both come out exactly right. And if the warehouse is far, put a medium shelf (L2) in between: now the far trip happens on almost no requests (two small fractions multiplied), and the average barely notices the warehouse at all.

Recall Quick self-check

Why does $H T$ have no $m$ in front? ::: You check the cache on every access, so hit time is paid unconditionally; it factors out to a flat $+ H T$ . What does the slope of the AMAT-vs- $m$ line equal? ::: The miss penalty $M P$ . At $m = 1$ what is AMAT? ::: $H T + M P$ — the full miss path, every access. Which miss rate goes inside the nested L2 parentheses? ::: The local L2 miss rate. Why does L2 shrink the DRAM cost so much? ::: The DRAM detour is now weighted by the product $m_{L 1} \times m_{L 2}$ , a tiny number.

Connections

Parent: Average memory access time (AMAT)
Cache Miss Rate & Miss Types (3 Cs) — where $m$ comes from
Miss Penalty & Main Memory Latency — what $M P$ measures
Multi-level Cache Hierarchy — Step 8's nesting
Cache Associativity & Hit Time tradeoff — why $H T$ isn't free to shrink
Locality of Reference — why $m$ is small
CPU Performance Equation — where AMAT feeds into runtime