Average memory access time (AMAT): Foundations — Hardware deep dive

Before you can read the parent note's formula $AMAT = H T + m \cdot M P$ , you must be able to point at every letter and say what it is. That is this page's only job. We build them in order, each one leaning on the one before.

0. The pictures we're working inside

Everything below lives on one mental image: a CPU (the brain) that keeps asking "give me the data at this address," and a stack of memories getting bigger but slower as you go down.

Look at the ladder above. The CPU is at the top. Each rung is a place data might live: a tiny fast cache near the top, a bigger slower one below, then the huge slow main memory (DRAM) at the bottom. Every symbol we define is a stopwatch reading or a counting-fraction somewhere on this ladder. See Multi-level Cache Hierarchy for the full ladder story.

1. Memory access — the event we are timing

Picture: one arrow leaving the CPU, going down the ladder until it finds the data, then coming back up. The length of that round trip (in time) is what we care about.

Why the topic needs it: AMAT is a per-access average. If we didn't fix "what counts as one access," averaging would be meaningless. This is our unit of counting. (Locality of Reference explains why consecutive accesses tend to land in the same fast rung — that's what keeps the average low.)

2. Cache, hit, and miss — did we find it up top?

Picture (above): two coloured paths. The green path (hit) is short — the CPU asks the cache, gets the answer, done. The red path (miss) is long — the cache shrugs, so the request continues down to slow memory and back.

Why the topic needs it: these two outcomes are the only two things that can happen on an access. AMAT is literally the average over exactly these two cases. Miss types (why misses happen) are catalogued in Cache Miss Rate & Miss Types (3 Cs).

3. Time symbols — the three stopwatches

Now we attach numbers to those paths. Each is a duration, measured in nanoseconds (ns) or CPU cycles.

Picture: on the miss path, the first segment (cache check) has length $H T$ ; the rest of the journey has length $M P$ . Total miss time $= H T + M P$ .

Why the topic needs it: the whole formula is built from these two durations. The reason $H T$ appears alone (un-multiplied) in the final formula is that you pay the green check on every access, hit or miss — it is unavoidable.

4. Miss Rate — the fraction that go slow

To average, we need to know how often the slow case happens. That's a fraction.

Picture (above): a bar of 100 accesses. A thin red slice is the misses ( $m$ ), the big green rest is the hits ( $1 - m$ ). If $m = 0.05$ , only 5 of every 100 requests take the long path — but as AMAT will show, those 5 can dominate your wait.

Why the topic needs it: $m$ is the weight on the slow case. Multiplying $M P$ by $m$ turns "the penalty if you miss" into "the penalty you pay on average per access."

5. Average / Expected value — the machine that combines them

This is the one piece of maths behind the whole topic. AMAT stands for average memory access time, so we must be crystal clear on what "average" means here.

Picture: imagine 100 coins, $p \times 100$ of them stamped " $a$ " and $q \times 100$ stamped " $b$ ." Add up all the stamps and divide by 100 — that's the average. Common outcomes pull the average toward their value; rare-but-huge outcomes can still pull it up noticeably.

Why it's the right tool (and not, say, just $(a + b) /2$ ): a plain midpoint would pretend hits and misses happen equally often. They don't — hits are common, misses rare. The weighted average respects those frequencies, which is the only honest way to say "what do I wait on average."

6. Subscripts and levels — reading $H T_{L 1}$ , $m_{L 2}$

The parent note stacks caches (L1, then L2). Subscripts just say which rung of the ladder.

Picture: water pouring down the ladder. All water hits L1; only the leaked fraction $m_{L 1}$ reaches L2. "Local" measures leakage at that rung relative to what arrived; "global" measures leakage relative to the original full pour at the top.

Why the topic needs it: the recursive formula $H T_{L 1} + m_{L 1} (H T_{L 2} + m_{L 2} M P_{L 2})$ uses the local rate inside, because each cache only ever sees the accesses forwarded to it. This distinction is the trickiest bit of multi-level tuning; getting local/global backwards is a classic bug.

Prerequisite map

Read it bottom-up: an access produces a hit or miss; those give us the two times and the fraction; a weighted average of them is AMAT; then subscripts and local/global let us nest it across levels.

A number to anchor it all

Picture (above): the parent's Example 1 — $H T = 1$ ns, $m = 0.05$ , $M P = 100$ ns. The bar shows AMAT $= 6$ ns split into the 1 ns you always pay (hit) and the 5 ns contributed by the rare misses ( $0.05 \times 100$ ). Notice the shocker: 5 of the 6 ns come from misses that are only 5% of accesses. That is why the whole topic obsesses over $m$ .

Equipment checklist

Cover the right side and test yourself. If any answer is fuzzy, re-read that section before the parent note.

What is "one memory access"?

One CPU request for the data at a single address; the thing AMAT times.

What is a cache hit?

The data was found in the cache — the short, fast path.

What is a cache miss?

The data was not in the cache, so we go further down the slower ladder.

What does

H T

(hit time) measure?

Time to check the cache and deliver data on a hit — paid on every access.

What does

M P

(miss penalty) measure?

The extra time beyond the hit check to fetch the block from the next level.

Total time on a miss?

H T + M P

— you still paid the hit check first, then the extra trip.

What is the miss rate

m

The fraction of accesses that miss, a number in

[0, 1]

; equals misses ÷ total accesses.

What is the hit rate?

1 - m

, the fraction of accesses that hit.

What is a weighted average?

p \cdot a + q \cdot b

— each outcome's cost weighted by how often it happens.

Why isn't

H T

multiplied by

m

in AMAT?

Because both hits and misses pay

H T

, so it factors out to a bare

+ H T

Local miss rate of L2?

L2 misses ÷ accesses that reached L2.

Global miss rate of L2?

L2 misses ÷ all CPU accesses

= m_{L 1} \times m_{L 2, local}

Which miss rate goes inside the recursive AMAT?

The local rate — each level only sees what's forwarded to it.

Connections

Average memory access time (AMAT) — the parent this page equips you for.
Cache Miss Rate & Miss Types (3 Cs) — where $m$ comes from.
Miss Penalty & Main Memory Latency — what makes $M P$ large.
Multi-level Cache Hierarchy — the ladder and its rungs.
Cache Associativity & Hit Time tradeoff — tuning $H T$ vs $m$ .
CPU Performance Equation — where AMAT feeds overall speed.
Locality of Reference — why hit rates are high at all.