Calling conventions and ABI: Worked examples — Hardware deep dive

We use the System V AMD64 ABI (Linux/macOS x86-64) as the concrete model, exactly as the parent did. Before anything, one refresher so no symbol is unearned:

Definition The pieces we will use (all from the parent)

Register: a tiny named storage box inside the CPU. See Registers and Register File. The integer-argument boxes, in order, are rdi, rsi, rdx, rcx, r8, r9; the answer comes back in rax.
Stack: a region of memory that grows downward (toward smaller addresses). rsp (the stack pointer) always points at the current top. See Stack and Stack Frames.
Caller-saved register: the called function may overwrite it freely. Callee-saved register (rbx, rbp, r12–r15): the called function must hand it back unchanged.
Float registers: xmm0–xmm7 carry floating-point arguments; xmm0 carries a floating-point return.
16-byte alignment rule: at the moment call is executed, rsp must be a multiple of 16; so inside the callee (after call pushed 8 bytes) rsp \equiv 8 \pmod{16}.

The scenario matrix

Every calling-convention problem is one of these cells. The examples below are labelled with the cell they cover.

#	Case class	What makes it tricky	Example
A	Few args, no locals	Baseline — nothing spills, nothing saved	Ex 1
B	More than 6 int args	Args 7+ spill to the stack, right-to-left	Ex 2
C	Mixed int + float args	Two separate counters: GP regs and XMM regs	Ex 3
D	Callee-saved discipline	Must `push`/`pop` a non-volatile register	Ex 4
E	Recursion / nested frames	Each call carves its own frame; return chain	Ex 5
F	Alignment (degenerate stack)	One extra `push` breaks `movaps`; padding fix	Ex 6
G	Zero args, void return	Degenerate: no arg regs, `rax` undefined	Ex 7
H	Small struct by value	Struct ≤ 16 B split across two registers	Ex 8
I	Word problem (real bug)	ABI mismatch causes silent corruption	Ex 9
J	Exam twist	Which register holds a value mid-call?	Ex 10

Ex 1 — Cell A: baseline, everything in registers

Forecast: guess the three registers before reading on.

Match args to the ordered list. a→rdi, b→rsi, c→rdx. Why this step? The order rdi, rsi, rdx, … is fixed by convention so the callee — written before it knows any caller — always looks in the same boxes.
Compute in the return register. add3 puts a+b then +c into rax (its 32-bit half eax). Why this step? The caller only reads rax for an integer return; putting the answer anywhere else means the caller reads garbage.
No stack frame. 3 args ≤ 6, no locals ⇒ nothing spills, rsp never moves. Why this step? Touching memory is slower than registers; the convention is designed so small functions never do.

Verify: $10 + 20 + 30 = 60$ . The caller reads eax and gets 60. ✓

Ex 2 — Cell B: seventh argument spills to the stack

Forecast: we only have 6 int registers — where does argument #7 go?

Fill the six registers. a→rdi, b→rsi, c→rdx, d→rcx, e→r8, f→r9. Why this step? That exhausts the integer-register budget of 6.
Push overflow args right-to-left. The caller pushes g (the only overflow) onto the stack before the call. Why this step? Right-to-left pushing means the leftmost overflow argument ends up at the lowest address — a fixed layout the callee can index into.
Locate g inside the callee. At entry, rsp points at the return address (8 bytes, pushed by call). g sits just above it, so the callee reads [rsp+8]. Why this step? The callee must account for the return address that call inserted between it and the pushed argument.

Look at the stack figure — the return address is the chalk-blue box, g is the pink box just above it.

Verify: function returns g = 7, read from [rsp+8], placed in rax. ✓

Ex 3 — Cell C: mixed integer and floating-point arguments

Forecast: do the ints and doubles share one counter, or two?

Run two independent counters. Integers consume rdi, rsi, …; doubles consume xmm0, xmm1, …. They do not interleave into one list. Why this step? Integer and float values live in physically different register files, so the ABI keeps two separate cursors.
Assign. a→rdi (1st int), x→xmm0 (1st float), b→rsi (2nd int), y→xmm1 (2nd float). Why this step? Each argument advances only its own counter.
Return. A double return comes back in xmm0, not rax. Why this step? Float results also travel in the float register file.

Verify: counters end at 2 ints (rdi, rsi) and 2 floats (xmm0, xmm1) — matches the 2 ints + 2 doubles in the signature. ✓

Ex 4 — Cell D: callee-saved discipline

Forecast: rbx is callee-saved — what does that force g to do?

Save on entry. push rbx before using it. Why this step? rbx is non-volatile: the caller trusts its value survives the call. To use it, g must first stash the caller's copy on the stack.
Do the work. rbx accumulates $s = 0 + 1 + 2 + \dots + (x - 1)$ . Why this step? This is exactly $\frac{x ( x - 1 )}{2}$ , the sum of the first $x - 1$ integers.
Restore on exit. Move rbx→rax (the return), then pop rbx to give the caller its value back. Why this step? Breaking the promise corrupts the caller silently — the parent's Example 3 warning.

Verify: for x=5, $s = 0 + 1 + 2 + 3 + 4 = 10 = \frac{5 \cdot 4}{2}$ . Return in rax = 10, rbx restored. ✓

Ex 5 — Cell E: recursion, nested frames

Forecast: how many return addresses are on the stack at the deepest point?

Each call carves a frame. fac(4) calls fac(3) calls fac(2) calls fac(1). See Recursion and Activation Records. Why this step? Each invocation needs its own n and its own return address; the stack (growing down) gives every call fresh space that can't collide.
Count the return addresses. At the deepest point (fac(1)), four calls have executed, so four return addresses are stacked. Why this step? call pushes exactly one return address per invocation; nothing has returned yet.
Unwind, multiplying. fac(1)=1, then 2·1=2, 3·2=6, 4·6=24, each result flowing back through rax. Why this step? Each ret pops its return address and hands its rax up to the caller's n * ….

Look at the figure — each nested frame is one chalk box; the arrows show rax climbing back out.

Verify: $4! = 4 \cdot 3 \cdot 2 \cdot 1 = 24$ , returned in rax. ✓

Ex 6 — Cell F: the alignment trap (degenerate stack)

Forecast: count bytes pushed and check the alignment rule.

Track rsp mod 16. At g's entry, rsp \equiv 8 \pmod{16} (the golden rule: call pushed 8 bytes onto a 16-aligned stack). Why this step? Alignment is a running invariant; you audit it byte by byte.
The push rbx shifts it. After push rbx, rsp \equiv 8 - 8 = 0 \pmod{16}. Why this step? Every 8-byte push subtracts 8 from rsp, flipping the low nibble between 0 and 8.
The inner call breaks it. Now calling the helper pushes another 8-byte return address: helper enters with rsp \equiv 0 - 8 = 8 \pmod{16} — but the helper's own body then wants movaps at 16-byte boundaries and finds rsp off by 8. A single extra sub rsp, 8 (or a second push) restores rsp \equiv 8 at its entry so its aligned locals land on multiples of 16. Why this step? movaps faults hard on misalignment; the compiler pads frames with exactly one 8-byte gap to keep the invariant.

Verify: entry 8; after one push 0; the fix sub rsp,8 gives 0-8 \equiv 8 \pmod{16}, restoring the required entry value. ✓

Ex 7 — Cell G: zero args, void return (degenerate)

Forecast: trick question — how many arg registers for zero args?

No argument registers touched. Zero args ⇒ rdi … r9 are left holding whatever the caller had. Why this step? The convention only reserves registers per argument; with no arguments it reserves none.
rax is undefined. A void return means the caller must not read rax — it may hold anything, and beep may have clobbered it (it is caller-saved). Why this step? No return value means no promise about rax; assuming a value here is a classic bug.

Verify: contract check — args used = 0, rax unspecified. ✓ (nothing numeric to compute)

Ex 8 — Cell H: small struct passed by value

Forecast: does a 16-byte struct go on the stack, or in registers?

Classify the struct. It is exactly 16 bytes of integer fields ⇒ it fits in two integer registers. Why this step? SysV AMD64 splits a small (≤ 16 B) struct field-by-field into registers rather than pushing it to memory — faster.
Assign the two eightbytes. p.x→rdi, p.y→rsi. Why this step? The first 8 bytes go in the first free GP register, the next 8 in the second.
Return the scalar. sumP computes x+y into rax. Why this step? The long result travels in rax like any integer.

Verify: $3 + 4 = 7$ in rax; struct consumed rdi, rsi (2 registers = 16 bytes / 8). ✓

Ex 9 — Cell I: real-world word problem (silent ABI bug)

Forecast: if the calling convention matches, why does it break?

Isolate the mismatch. The calling convention (which registers, who saves what) is identical — so linking finds no error. Why this step? Linkers match symbol names, not data-layout agreements. See Name Mangling in C++ for how names alone can hide type differences.
Name the real contract broken. Data type size/alignment is part of the ABI, not the calling convention. One side reads 10 bytes where the other wrote 16. Why this step? The parent's steel-man: calling convention ⊂ ABI. Two objects can share a convention yet disagree on long double width.
Consequence. Bytes are misread → silent numeric corruption, no crash, no linker warning. Why this step? Nothing checks byte layout at link time, so the failure surfaces only in wrong output.

Verify: contract check — convention identical, sizeof(long double) differs (10 vs 16 bytes) ⇒ ABI mismatch. ✓

Ex 10 — Cell J: exam twist ("which register mid-call?")

Forecast: whose job was it to preserve rcx?

Classify rcx. rcx is caller-saved (volatile): the callee may clobber it freely. Why this step? The parent's list: caller-saved = rax, rcx, rdx, rsi, rdi, r8–r11.
Locate the responsibility. Because rcx is caller-saved, A (the caller) must save it before the call if it needs it afterward — B owes nothing. Why this step? The whole point of the split is to avoid wasted saves; volatile registers are the caller's problem.
The fix. A should push rcx before call B and pop rcx after — or keep the value in a callee-saved register (rbx, r12–r15) instead. Why this step? Either strategy makes the value survive: save-around-call, or park it where the callee promises to preserve it.

Verify: rcx ∈ caller-saved set ⇒ preserving it is the caller's duty. A omitted it ⇒ correct diagnosis. ✓

Recall Which cell was each example?

Ex1 A · Ex2 B · Ex3 C · Ex4 D · Ex5 E · Ex6 F · Ex7 G · Ex8 H · Ex9 I · Ex10 J.

Recall Quick self-test

7th integer argument lives where? ::: On the stack at [rsp+8] on entry (return address is at [rsp]). A double argument and an int argument — do they share a register counter? ::: No: ints use rdi…, floats use xmm0…, two separate counters. void function — is rax meaningful? ::: No, undefined; never read it. rcx clobbered across a call — whose fault? ::: The caller's; rcx is caller-saved, so the caller must preserve it. Same calling convention but long double size differs — what breaks? ::: The ABI (data sizing), causing silent corruption; the linker won't catch it.

See also: x86-64 vs ARM64 AAPCS for how these cells change on ARM, and System Calls and Syscall Interface for the kernel-boundary variant of argument passing.