Calling conventions and ABI: Question bank — Hardware deep dive

Definition Vocabulary this page relies on (defined here, not linked away)

ABI (Application Binary Interface) — the full binary-level rulebook two compiled pieces of code must share: argument passing, register save rules, type sizes/alignment, struct layout, name mangling, object file format, and syscall numbers.
API (Application Programming Interface) — the source-level contract: function names and signatures. The API is what you read; the ABI is what survives after compilation into machine code.
SysV AMD64 — the "System V AMD64" ABI, the calling convention used by Linux and macOS on 64-bit Intel/AMD chips. It is our default model on this page.
Microsoft x64 — the different calling convention Windows uses on the same hardware (see the quadrant figure below). Same CPU, different rulebook.
AAPCS — the "ARM Architecture Procedure Call Standard": ARM64's calling convention, structurally similar but with its own register names. See x86-64 vs ARM64 AAPCS.
Red zone — a small 128-byte scratch area just below rsp that SysV lets a leaf function use without moving rsp. Defined and pictured below.

Two visual anchors before the questions — study these, then the Q&A refers back to them.

The stack frame above shows, from high address downward: incoming stack arguments (7th+), the return address that call pushed, saved callee-saved registers, local variables, and the red zone below rsp. Every stack-related question below points at a slot in this picture.

The register table above is the "who saves what" split — caller-saved (volatile) in orange, callee-saved (non-volatile) in violet — including the floating-point xmm registers, which the questions ask about.

The quadrant above compares SysV AMD64 (Linux/macOS) with Microsoft x64 (Windows) on the same CPU — same hardware, deliberately different rules.

True or false — justify

Every argument is always passed in a register.

False. Only the first few (6 integer/pointer args in SysV AMD64 — see figure s02) go in registers; extra ones spill onto the stack into the slots at the top of figure s01. See Stack and Stack Frames.

The CPU hardware knows what a "function argument" is.

False. The CPU only has registers and memory; "argument" is a pure software convention agreed on in advance — the hardware never sees the concept.

Two .o files that share a calling convention are guaranteed to link and run correctly together.

False. They must also agree on the rest of the ABI — type sizes, struct layout, alignment, name mangling. A shared calling convention is necessary but not sufficient. See Linkers and Object File Format.

If a function never uses rbx, it still must push and pop it.

False. Callee-saved discipline says "save it only if you use it." A function that never touches rbx leaves it alone — no save needed.

Caller-saved registers are safer to use than callee-saved ones.

False — they aren't "safer," just governed by a different rule. A caller-saved value survives a call only if the caller itself saves it first; the callee may freely clobber it (orange rows in figure s02).

The call instruction is a primitive the hardware provides specifically for functions.

Partly true but misleading. call is just push return_address; jmp target — a convenience combining two primitives. The function abstraction is still convention, not hardware.

The return address lives in a register.

False in SysV AMD64 — call pushes it onto the stack (the labelled slot in figure s01). Some ISAs like ARM64 (AAPCS) instead put it in a link register x30 — see x86-64 vs ARM64 AAPCS.

Floating-point arguments share the same registers as integer arguments.

False. Integers use rdi/rsi/rdx/rcx/r8/r9; floats use a separate pool xmm0–xmm7. They are counted independently.

In SysV AMD64 the xmm floating-point registers are all callee-saved.

False. In SysV all of xmm0–xmm15 are caller-saved (volatile) — the callee may clobber any of them, so the caller must save a live float across a call. (Contrast: Microsoft x64 makes xmm6–xmm15 callee-saved — see figure s03.)

The ABI is a source-code concept like the API.

False. The API is source-level (names and signatures); the ABI is the binary-level contract that survives compilation.

Spot the error

"To use rbx inside my function, I push rbp at entry and pop it before ret."

Wrong register saved. If you clobber rbx you must save rbx itself — the ABI's promise is per register: the caller trusts each named callee-saved register individually. Saving rbp leaves the caller's rbx corrupted.

"7 arguments? I'll put the 7th in r10 since it's the next register after r9."

Wrong. The ABI fixes the argument sequence at exactly rdi/rsi/rdx/rcx/r8/r9 so the callee — compiled with no knowledge of the caller — knows precisely where to look. r10 is defined as caller-saved scratch, not an argument slot, so the callee would never read it. The 7th arg goes on the stack.

"The callee reads the 7th stack argument from [rsp]."

Wrong offset. call pushed the return address first, so at entry rsp points at that, not the argument. The ABI lays the 7th argument just above the return address, giving ==[rsp+8]== (see the two adjacent slots in figure s01).

"I subtracted 8 from rsp for one local, so alignment is fine."

Likely broken. Algebra: outside a call rsp ≡ 0 (mod 16); call pushes 8 bytes so at entry rsp ≡ 8 (mod 16); subtracting 8 gives 0 (mod 16), but the next call you make pushes another 8 and re-misaligns it. You must size the frame so rsp ≡ 0 (mod 16) right before each call — see the WHY question below.

"My function returns two values, so I return one in rax and one in rbx."

Wrong register. The ABI defines a 128-bit result as the pair rdx:rax precisely because both are caller-saved scratch — free for the callee to overwrite. rbx is callee-saved, i.e. reserved to hold the caller's state across the call, so returning through it would clobber a value the caller trusts.

"The caller sets up the arguments after executing call."

Wrong order. Arguments must be in place before call, because call transfers control to the callee immediately — afterwards the callee is already reading rdi/rsi/….

Why questions

Why split registers into caller-saved and callee-saved instead of one rule?

A pure caller-saves rule wastes saves on registers the callee never touches; a pure callee-saves rule wastes saves on registers the caller doesn't need afterward. Splitting (figure s02) lets each side save only what's actually at risk.

Why does SysV AMD64 pass exactly the first 6 args in registers?

It's an empirical sweet spot — most functions have ≤ 6 arguments, so most calls never touch memory for args, giving a big speed win. See Registers and Register File.

Why must rsp be 16-byte aligned before a call?

Step by step: (1) SSE instructions like movaps fault unless their memory operand is 16-byte aligned. (2) The compiler places aligned locals relative to rsp. (3) Since call pushes an 8-byte return address, the ABI requires rsp ≡ 0 (mod 16) before call so that inside the callee rsp ≡ 8 (mod 16), and the callee can then subtract a padded frame back to a multiple of 16. Guaranteeing this once at the boundary makes every aligned local free.

Why are stack arguments pushed right-to-left?

So the leftmost overflow argument ends up at the lowest address, giving the callee a predictable, ascending order to read them from (top slots of figure s01).

Why does recursion need the stack rather than registers?

Each nested call needs its own copy of locals and return address; registers are a fixed finite set and would be overwritten, so each activation gets a fresh stack frame. See Recursion and Activation Records.

Why can't a system call use the same register order as an ordinary function call?

The kernel defines its own convention (e.g. syscall number in rax, args in rdi…, and r10 replaces rcx because the syscall instruction clobbers rcx). See System Calls and Syscall Interface.

Why does C++ need name mangling on top of the C ABI?

Overloading and namespaces mean several functions share a source name; the linker needs distinct binary symbols, so the compiler encodes types into the name. See Name Mangling in C++.

Why does Windows use a different x86-64 convention than Linux at all?

The two operating systems standardized independently; the hardware imposes no convention, so each OS picked its own (4 arg registers on Windows vs 6 on SysV, different volatile sets — figure s03). This is exactly why a Linux .so won't link against a Windows .dll.

Edge cases

A function with zero arguments — does the convention still apply?

Yes. It simply uses no argument registers, but it still owes callee-saved discipline, stack alignment, and returning in rax if it has a result.

A function that never returns (e.g. calls exit) — must it still preserve callee-saved registers?

No practical obligation, since control never comes back to the caller. The preservation promise only matters if the caller resumes.

A leaf function (calls nothing) with few locals — does it need a stack frame?

Often not. SysV gives every function a red zone: 128 bytes below rsp (the shaded region at the bottom of figure s01) that a leaf function may use as scratch without moving rsp at all. It works only for leaves because any call would let the callee overwrite that region. This saves the sub rsp / add rsp pair entirely.

Passing a struct larger than 16 bytes by value — where does it go?

The ABI classifies it as MEMORY, so the caller copies it into the stack argument area (top of figure s01), not into registers, because it exceeds the two-register class limit. If a return value is that large, the caller instead passes a hidden pointer in rdi (called sret, "structure return") pointing to space it allocated, and the callee writes the result there — because there is no register wide enough to carry it back.

A recursive call at depth 10000 — what convention property makes it work at all?

Each call carves a fresh frame downward from rsp, so activations never collide — until the stack runs out (stack overflow), the boundary case where the invariant meets a hard memory limit.

Recall One-line self-test

Cover everything: can you state who saves which register and why the split exists? ::: Caller saves volatile registers it still needs; callee saves any non-volatile register it uses — the split minimizes total memory traffic by making each side save only what's actually at risk.