Calling conventions and ABI

WHAT is an ABI vs a Calling Convention?

HOW arguments and return values travel

Using the System V AMD64 ABI (Linux/macOS x86-64) as our concrete model:

The Stack, and the "who saves what" problem

DERIVATION: building a function call from first principles

We have no call semantics — only: registers, memory, a stack pointer rsp, and an instruction pointer rip. Let's invent the convention.

Step 1 — How does the callee know where to return? Why this step? The callee finishes and must resume the caller's next instruction. So the caller must record the return address somewhere the callee can find it. Decision: push it onto the stack. The call instruction does exactly this: $\texttt{call } f \;\equiv\; \texttt{push rip\_next};\ \texttt{jmp } f$

Step 2 — Where are the arguments? Why this step? Callee code is written before it knows any specific caller, so the location must be fixed by convention, not by the caller's whim. Decision: rdi, rsi, … in order.

Step 3 — How does the callee get its own scratch/local space? Why this step? Locals need memory that survives the function and can't collide with the caller's frame. Decision: subtract from rsp to carve a stack frame downward: $\texttt{rsp} \leftarrow \texttt{rsp} - (\text{frame size})$

Step 4 — Returning the answer and cleaning up. Why this step? The caller expects the result in a fixed place (rax) and expects the stack restored. Decision: $\texttt{mov rax, result};\ \texttt{add rsp, frame};\ \texttt{ret}$ where ret ≡ pop rip.

That's the entire convention — derived, not memorized.

Worked Example 1 — reading a call in assembly

Source:

int add3(int a, int b, int c) { return a + b + c; }
int main() { return add3(10, 20, 30); }

Assembly for the call:

main:
    mov edi, 10      ; a → rdi   (Why? 1st int arg)
    mov esi, 20      ; b → rsi   (Why? 2nd int arg)
    mov edx, 30      ; c → rdx   (Why? 3rd int arg)
    call add3        ; Why? pushes return addr, jumps
    ret              ; result already in eax, return it
add3:
    lea eax, [rdi+rsi]   ; Why? add first two, put in return reg rax
    add eax, edx         ; Why? add third
    ret                  ; Why? pop return address, resume main

Why no stack use? Only 3 args (≤ 6) and no locals — everything lives in registers. Fast.

Worked Example 2 — argument spilling to the stack

long f(long a,long b,long c,long d,long e,long f,long g) { return g; }

Why this matters: 7 args > 6 registers. Args 1–6 → rdi…r9; arg g (7th) is passed on the stack.

; caller pushes g first (right-to-left)
mov  rdi,...  ; a ... through r9 for f
mov  rax,[rsp+8]  ; callee reads g from stack (above return addr)
ret

Why [rsp+8]? At entry rsp points at the return address; the 7th arg sits just above it.

Worked Example 3 — callee-saved register discipline

long g(long x){ long s=0; for(long i=0;i<x;i++) s+=i; return s; }

g:
    push rbx          ; Why? we WANT to use rbx (callee-saved) → must save it
    xor  rbx, rbx     ; s = 0
    ...
    mov  rax, rbx     ; return s
    pop  rbx          ; Why? restore the caller's rbx before returning
    ret

Why bother? rbx is callee-saved: the caller trusts it's unchanged after the call. Breaking this promise corrupts the caller silently.

Flashcards

Connections

• Stack and Stack Frames
• Registers and Register File
• System Calls and Syscall Interface
• Linkers and Object File Format
• Recursion and Activation Records
• x86-64 vs ARM64 AAPCS
• Name Mangling in C++

Concept Map

Hinglish (regional understanding)