Calling conventions and ABI: Visual walkthrough — Hardware deep dive

Before we begin, three words that appear in every picture — defined here, in plain language, so no symbol is ever used before you have seen it.

Definition The three things our bare machine actually has

Register — a tiny, named storage box inside the CPU. Holds one number. We will see boxes named rdi, rax, rsp, rip. (More in Registers and Register File.)
Stack — a region of ordinary memory that we use like a pile of plates. It grows downward: "push" carves out a lower address, "pop" gives it back. (More in Stack and Stack Frames.)
==Stack pointer rsp== — a register whose number is an address: it always points at the top plate of the stack (the lowest used address). Move rsp = move the top of the stack.
==Instruction pointer rip== — a register holding the address of the next instruction the CPU will run. Change it, and the CPU jumps.

That is the entire toolbox. Everything below is invented out of just these four boxes plus memory.

The picture below is our starting laboratory — memorise its layout, because every later figure is this same layout in motion.

Step 1 — The problem: two strangers must meet at one address

WHAT. We have a caller (call it main) and a callee (f). main reaches the moment it wants f to run, then wants to continue at its own next instruction afterward.

WHY this is a problem. The CPU only ever does "run the instruction rip points at, then advance". It has no concept of "go do f, then come back". Coming back means: someone must have remembered the address to return to. main is the only one who knows that address — so main must record it before leaving.

PICTURE. In the figure, rip inside main is about to leave. The amber question mark is the return address that will be lost forever the moment we jump — unless we save it somewhere f can find it.

We need a fixed, agreed hiding place for that address. Not a place main picks at whim (f is compiled separately and cannot read main's mind) — a place fixed by convention. The natural choice is the top of the stack.

Step 2 — Inventing `call`: push the return address, then jump

WHAT. We define one combined action:

$one instruction call f \equiv save where to come back push rip_next; go run f jmp f$

Term by term:

$rip_next$ — the address of the instruction right after the call. This is the "come back here" note.
$push$ — do two micro-steps: $rsp \leftarrow rsp - 8$ (carve 8 bytes lower, since an address is 8 bytes on a 64-bit machine), then write $rip_next$ into that new top slot.
$jmp f$ — set $rip \leftarrow$ address of f. The CPU now runs f.

WHY. Pushing onto the stack solves the "fixed hiding place" problem: f doesn't need to know which caller — it only needs to know the return address is on top of the stack. That single rule works for every caller in the universe.

PICTURE. Watch rsp step down by 8 and the amber return address land in the freshly-opened slot. rip swings over to f.

Step 3 — Where are the arguments? Registers, by fixed name

WHAT. main wants to hand f some numbers (its arguments). We decide, once and for all, the order of boxes they go in. Using the System V AMD64 model:

$1^{st} rdi, 2^{nd} rsi, 3^{rd} rdx, 4^{th} rcx, 5^{th} r8, 6^{th} r9$

WHY registers, and WHY a fixed order? Registers are the fastest storage the CPU owns — reading rdi costs nothing extra, reading memory costs a trip out of the core. And the order must be fixed by agreement because f is compiled without ever seeing main; the only way f can "find argument #1" is if everyone forever agrees it lives in rdi.

WHY only 6? It is an empirical sweet spot: the vast majority of real functions take 6 or fewer arguments, so those calls never touch memory for arguments at all — pure register speed.

PICTURE. The caller fills the six named boxes left-to-right; the callee will read those same boxes by name.

Step 4 — Carving local scratch space: the stack frame

WHAT. f may need its own temporary memory — locals, spilled values. It carves them by moving rsp down:

$rsp \leftarrow rsp - frame size in bytes N$

WHY subtract, WHY the stack? Subtracting from rsp opens $N$ bytes below the return address — a private room that cannot collide with main's memory (that's all at higher addresses, untouched). It is automatically freed later by moving rsp back up. This private room is called a stack frame (deep dive: Recursion and Activation Records).

PICTURE. The frame grows downward as an amber-outlined block; main's frame above is greyed and safe.

Step 5 — The alignment invariant: why `rsp ≡ 8 (mod 16)` at entry

WHAT. There is a numeric rule about rsp's value the instant f begins:

$rsp \equiv 8 (mod 16)$

Read $\equiv 8 (mod 16)$ as: "when you divide rsp by 16, the remainder is 8" — i.e. rsp is 8 more than a multiple of 16.

WHY this exact number? The convention says: just before the call, rsp must be a clean multiple of 16 (16-byte aligned). Then call pushed 8 bytes (the return address, Step 2). Aligned‑to‑16 minus 8 = "8 above a multiple of 16". So at entry the remainder is exactly 8 — a fingerprint proving alignment was respected.

WHY care at all? Fast vector instructions like movaps demand that the memory they touch be 16‑byte aligned. If the frame is misaligned, they don't just run slow — they segfault. The compiler pads the frame size $N$ (Step 4) precisely to keep this invariant true.

PICTURE. A ruler along the stack marks multiples of 16; the return-address slot lands at the "+8" tick, and every well-formed frame keeps that offset.

Step 6 — Returning the answer and undoing everything

WHAT. f finishes. Three obligations, in order:

$(a) answer into agreed box mov rax, result; (b) give back the frame add rsp, N; (c) pop rip ret$

Term by term:

(a) the result goes into rax — the fixed "answer box" the caller will read. (Floating results use xmm0; 128-bit results use rdx:rax.)
(b) $add rsp, N$ raises rsp back up by exactly the $N$ we subtracted in Step 4 — the private room is returned to the pile. After this, rsp again points at the return address, exactly as at entry.
(c) $ret \equiv pop rip$ : read the top-of-stack (the return address from Step 2) into rip, and add 8 to rsp. The CPU is now running main's next instruction, and the stack is byte-for-byte as it was before the call.

WHY this exact undo? A call must be transparent: after it returns, main should see its stack and its saved registers unchanged, plus the answer in rax. (b) restores the stack, (c) restores control flow, (a) delivers the payload.

PICTURE. Reverse of Steps 2–4 played backward: frame collapses up, return address pops into rip, rax glows with the answer.

Step 7 — Edge case A: more than 6 arguments (spill to the stack)

WHAT. With 7 arguments, args 1–6 fill the registers; the 7th spills onto the stack. The caller pushes overflow args right-to-left, so the leftmost overflow argument ends up at the lowest address, and call then pushes the return address on top.

WHY on the stack, WHY that layout? There are only six argument registers — the 7th has nowhere else to go. Right-to-left pushing makes the overflow args sit in natural left-to-right order in ascending memory, so the callee reads them predictably. Because the return address is pushed last (Step 2), it lands above the spilled args, and the 7th argument sits just past it:

$arg7 = [rsp + 8]$

Read: "8 bytes above the current top" — skip over the 8-byte return address, and there is arg 7.

PICTURE. The 7th argument (amber) sits one slot above the return address; the six registers are full.

Step 8 — Edge case B: the "who saves what" split

WHAT. f will use registers — maybe the very ones holding main's live values. Two families:

Caller-saved (volatile) — rax, rcx, rdx, rsi, rdi, r8–r11. f may freely overwrite these; if main still needs them, main saves them before the call.
Callee-saved (non-volatile) — rbx, rbp, r12–r15, rsp. If f wants to use one, f must save the old value on entry and restore it before ret.

WHY split them instead of one blanket rule? A pure caller-saves rule wastes work saving registers f never touches. A pure callee-saves rule wastes work saving registers main no longer cares about. Splitting the registers into two teams lets each side save only what is genuinely at risk — the minimum possible memory traffic.

PICTURE. Two coloured teams of registers; f wraps its use of a callee-saved box rbx in a push rbx … pop rbx bracket, handing it back unbroken.

Worked example The discipline in code

g:
    push rbx        ; save caller's rbx (callee-saved → my duty)
    xor  rbx, rbx   ; now safe to use rbx as scratch
    ...             ; work happens
    mov  rax, rbx   ; answer into the return box
    pop  rbx        ; hand caller's rbx back exactly as it was
    ret

Break the push/pop pairing and main's rbx is silently corrupted — the worst kind of bug, because g "looks" correct.

The one-picture summary

Everything above, on one canvas: the stack's life across a whole call. Read top-to-bottom as time. Registers carry args in and the answer out; the stack holds the return address and the frame; alignment stays at the +8 tick throughout.

Recall Feynman retelling — the whole walkthrough in plain words

A machine that only knows "run the next instruction" wants to borrow another chunk of code and come back. So the caller writes a sticky-note — "come back to here" — and slides it onto the top plate of a downward-growing pile of plates (that's call: push the return address, then jump). Everyone agrees, forever, that the first few numbers you hand a function live in six specific boxes (rdi, rsi, rdx, rcx, r8, r9); if you have more than six, the leftovers get stacked on plates just under the sticky-note. The function carves out its own private plates below for scratch work, always keeping the pile height at the same "off-by-8" tick so the fast vector instructions don't crash. When done, it drops the answer in the box everyone reads (rax), lifts its scratch plates back off, and peels the sticky-note off the top to jump home. Along the way, some boxes are "yours to trash" (caller-saved) and some are "borrow-and-return-unbroken" (callee-saved) — a split chosen so nobody ever saves a box that wasn't actually at risk. Nothing here was built into the CPU: it is all agreement, and that agreement is the calling convention at the heart of the linkable ABI.

Flashcards

Why must the return address be saved on the stack rather than in a fixed register chosen by the caller?

Because the callee is compiled separately and cannot know the caller; a fixed stack location (top of stack after call) works for every possible caller uniformly.

What two micro-steps does call f perform?

push rip_next (rsp ← rsp − 8, store return address), then jmp f (rip ← address of f).

After entry to a function, what does [rsp] hold?

The return address that call pushed.

Why is rsp ≡ 8 (mod 16) at function entry?

Because rsp was 16-byte aligned before call, and call pushed 8 bytes (the return address), leaving remainder 8.

Where does the 7th integer argument live, and how is it read?

On the stack, just above the return address, at [rsp+8] on entry.

Why split registers into caller-saved and callee-saved instead of one rule?

To minimise total saves — each side saves only the registers actually at risk (callee never-touched vs caller-no-longer-needed).