Calling conventions and ABI: Exercises — Hardware deep dive

Reference facts you may need (all from the parent note):

Level 1 — Recognition

Problem 1.1

For the call foo(a, b, c, d) where all four arguments are int, name the register that holds each argument, in order.

Recall Solution 1.1

The integer-argument order is fixed: rdi, rsi, rdx, rcx, r8, r9. We only need the first four slots.

a → rdi (1st slot)
b → rsi (2nd slot)
c → rdx (3rd slot)
d → rcx (4th slot)

WHY these and not others? The convention is a fixed lookup table so the callee — compiled before it ever meets a caller — knows exactly where to read.

Problem 1.2

long h(void) returns a single 64-bit integer. Which register carries that value back to the caller?

Recall Solution 1.2

The integer return register is rax. A single 64-bit value fits entirely in rax; rdx is only recruited when the value is 128 bits wide.

Problem 1.3

Classify each register as caller-saved or callee-saved: rax, rbx, r10, r12, rdi.

Recall Solution 1.3

Match against the two lists:

rax → caller-saved (return register, always volatile)
rbx → callee-saved
r10 → caller-saved
r12 → callee-saved
rdi → caller-saved (it's an argument register — the callee is free to overwrite it)

Level 2 — Application

Problem 2.1

Write the essential moves the caller makes for bar(7, 42) where both args are int, then the call. (Assume the values are literals.)

Recall Solution 2.1

Two args → slots rdi, rsi. Because they are 32-bit int, we load the 32-bit sub-registers edi/esi (writing a 32-bit register zero-extends into the full 64-bit register for free):

mov  edi, 7      ; 1st arg → rdi
mov  esi, 42     ; 2nd arg → rsi
call bar         ; push return addr, jump

Problem 2.2

A function long k(long a) needs to use r13 internally. What two instructions must it add (and where), and why?

Recall Solution 2.2

r13 is callee-saved, so k must leave it exactly as it found it.

k:
    push r13         ; save caller's r13 on entry
    ...              ; freely use r13 here
    pop  r13         ; restore before returning
    ret

WHY: the caller trusts r13 is unchanged across the call. push/pop are a matched pair — save on entry, restore just before ret.

Problem 2.3

Alignment. Just after the CPU executes call foo (which pushed the 8-byte return address), rsp holds the value 0x7fff_ffff_e2a8. Is the alignment invariant satisfied at foo's entry?

Recall Solution 2.3

The invariant is $rsp \equiv 8 (mod 16)$ at entry. Take the low hex digits: ...a8 → the byte value is 0xa8 = 168. $168 mod 16 = 8.$ Since $8 \equiv 8 (mod 16)$ , the invariant holds. Good — SSE instructions like movaps inside foo will not fault.

Level 3 — Analysis

Problem 3.1

Trace the argument locations for:

long f(long a, long b, long c, long d, long e, long g, long h);

Where does each argument live at f's entry?

Recall Solution 3.1

Seven long args, but only six integer registers.

a → rdi, b → rsi, c → rdx, d → rcx, e → r8, g → r9 (slots 1–6)
h (7th) → on the stack.

Extra args are pushed right-to-left, so the leftmost overflow arg (h here) ends up at the lowest stack address of the overflow region. At entry, rsp points at the return address, so h sits just above it: $h = [rsp + 8] .$

Problem 3.2

Study the diagram below (Figure 1). After call f runs, list the stack contents from high address to low, given f has a 32-byte local frame.

Recall Solution 3.2

Reading Figure 1 top (high address) → bottom (low address):

Overflow argument h (pushed by caller before call) — highest.
Return address (pushed by call itself) — this is where rsp points at entry.
Local frame, 32 bytes (sub rsp, 32 inside f) — lowest; new rsp points at its bottom.

Notice h lives above the return address ([rsp+8] at entry), and locals live below it. The stack grows downward (toward lower addresses) — the orange arrow.

Problem 3.3

Consider this buggy fragment. What silent bug does it contain?

compute:
    mov  rbx, rdi     ; stash input in rbx
    imul rbx, rbx     ; square it
    mov  rax, rbx     ; return value
    ret

Recall Solution 3.3

rbx is callee-saved. compute writes to rbx but never pushed it on entry nor popped it before ret. So it returns with the caller's rbx destroyed. The return value in rax is correct, which is exactly why the bug is silent — everything looks fine until the caller later reads its own (now-corrupted) rbx. Fix: either bracket with push rbx / pop rbx, or use a caller-saved scratch register such as r10/r11 and avoid saving entirely.

Level 4 — Synthesis

Problem 4.1

Given:

double scale(double x, long n, double y);

Assign every argument to its register. (Remember floats and ints use separate register banks.)

Recall Solution 4.1

Integer args and floating-point args are counted on two independent lists:

Floats fill xmm0, xmm1, xmm2, …
Integers fill rdi, rsi, rdx, …

Walk the signature left to right, dispensing from the correct bank each time:

x (double) → xmm0 (1st float)
n (long) → rdi (1st integer)
y (double) → xmm1 (2nd float)

Return type is double, so the result comes back in xmm0. Key insight: a long sandwiched between two doubles does not push the second double from xmm1 to xmm2 — the banks are counted separately.

Problem 4.2

A function long depth3() calls another function three times in sequence, and between the calls it must keep a running counter alive. Should the counter live in a caller-saved or callee-saved register? Justify with the total number of save/restore operations each choice costs.

Recall Solution 4.2

The counter must survive across three calls. A value in a caller-saved register would be clobbered by each callee, so depth3 would have to save-and-restore it around every call: $3 \times 2 = 6$ memory operations. Put it in a callee-saved register instead. Then:

depth3 saves it once on entry, restores once on exit: $2$ operations total.
The three callees each promise to preserve it for free.

Cost comparison: $6$ (caller-saved) vs $2$ (callee-saved). Choose callee-saved — this is exactly the reason the ABI splits registers into the two classes: values that live across calls belong in callee-saved registers.

Level 5 — Mastery

Problem 5.1

You write a leaf helper in hand-assembly that pushes exactly one 8-byte register on entry, does some work, then makes a nested call to printf. It segfaults inside printf at a movaps instruction. Diagnose the cause and give a one-instruction fix.

Recall Solution 5.1

Trace the alignment. Let $rsp \equiv 8 (mod 16)$ at your entry (the invariant).

push one register: rsp drops by 8 → now $rsp \equiv 0 (mod 16)$ .
You execute call printf: that pushes an 8-byte return address → at printf's entry $rsp \equiv 8 (mod 16)$ ... wait — recompute: $0 - 8 = - 8 \equiv 8 (mod 16)$ . That would be fine.

The real trap: the invariant printf relies on is that rsp is $8 (mod 16)$ at its entry — which requires rsp to be $0 (mod 16)$ immediately before your call printf. After one push, rsp is $0 (mod 16)$ , so call lands printf at $8 (mod 16)$ ✓. So a single push is actually safe. The segfault means you pushed an even number that returned you to $8 (mod 16)$ before the call (making printf enter at $0 (mod 16)$ — misaligned for its own SSE spills). Concretely, if you pushed two registers (or sub rsp, 16 plus a stray 8), you break it. One-instruction fix: insert sub rsp, 8 (or push one extra dummy 8-byte value) before call printf so that rsp is $0 (mod 16)$ at the call site; then add rsp, 8 after. This restores the invariant printf needs.

Problem 5.2

Two .o files are compiled by different compilers that share the identical SysV calling convention. Yet linking and running them together produces silent numeric corruption when one passes a long double to the other. Explain why the calling convention being identical does not save you, and name the layer at fault.

Recall Solution 5.2

The calling convention answers "which slot does an argument go in?" — but not "how many bytes is a long double?". Suppose compiler A treats long double as 80-bit (x87 extended) padded to 16 bytes, while compiler B treats it as 128-bit quad precision. Even if both agree to pass it in the same place (on the stack, 16-byte aligned), the bit-level meaning of those 16 bytes differs. B reads A's 80-bit-plus-padding pattern as a genuine 128-bit float → garbage. Layer at fault: the ABI, specifically its data-type size and representation clause — a strict superset of the calling convention. (See the parent's steel-man: calling convention ⊂ ABI.) This is why "same calling convention" is not the same as "ABI-compatible."

Problem 5.3

Design task. You must pass a struct Pair { long a; long b; } by value to a function. The ABI says small aggregates (≤ 16 bytes, all integer fields) are passed in registers, field by field. Where do a and b end up for void use(Pair p)? Then answer: where do they go for void use2(long z, Pair p)?

Recall Solution 5.3

Pair is 16 bytes, both fields integer, so it is not passed by pointer — it is flattened into two integer register slots.

use(Pair p): p.a → rdi (1st integer slot), p.b → rsi (2nd integer slot). Return-wise nothing; the struct occupies the first two integer registers.
use2(long z, Pair p): z consumes the 1st slot → rdi. Then the flattened struct consumes the next two slots: p.a → rsi, p.b → rdx.

Key idea: register slots are consumed in source order, and an aggregate expands into as many consecutive slots as it has eightbytes. This is why struct-by-value can be as cheap as passing two loose longs — no memory round-trip at all.

Recall Feynman recap

Question ::: What single fact resolves most of these exercises at once? Answer ::: The ABI is a fixed lookup table you walk in source order — integers dispense from rdi,rsi,rdx,rcx,r8,r9; floats from xmm0–7; overflow goes on the stack; the answer returns in rax (or xmm0); and callee-saved registers (rbx,rbp,r12–r15) must be handed back untouched. Every problem is just applying that table plus the $8 (mod 16)$ alignment rule.

Related deep material: x86-64 vs ARM64 AAPCS, Recursion and Activation Records, System Calls and Syscall Interface, Name Mangling in C++, Linkers and Object File Format.