System calls — user mode vs kernel mode, trap mechanism: Visual walkthrough

We are answering ONE question, slowly: how does a program that is forbidden from touching the disk end up getting the disk touched — safely?

Step 1 — Two piles of instructions, one CPU

WHAT. A CPU is a machine that reads instructions one after another. Some instructions are harmless (add two numbers). Some are dangerous (talk to the disk controller, turn off all interrupts, rewrite the memory map). Picture all instructions sorted into two piles: a big safe pile and a small dangerous pile.

WHY split them? Because your program is untrusted code. If a buggy app could run a dangerous instruction, it could wipe a disk or read another program's passwords. So the hardware needs a way to say "you may run the safe pile, but not the dangerous one."

PICTURE. The CPU holds a single mode bit — one bit, literally one binary digit — that says who is in charge right now. 0 = kernel (may run everything). 1 = user (dangerous instructions fault, i.e. the CPU refuses and jumps to the OS instead).

Figure — System calls — user mode vs kernel mode, trap mechanism

$CPL \in {0, 3}$

$CPL$ — the number the hardware reads before every dangerous instruction.
$0$ — kernel: the gate is open.
$3$ — user: the gate is shut; dangerous instructions trigger a fault instead of running.

Step 2 — The wall, and the one legal door in it

WHAT. Draw a vertical wall. Left of the wall: user space (your program's world, $CPL = 3$ ). Right of the wall: kernel space ( $CPL = 0$ ). Your program lives on the left and cannot walk through the wall.

WHY not just jump across? Because a normal jump (a call to some address) does not change the mode bit. If jumping alone flipped you to kernel, any program could jump into a privileged routine and own the machine. So the wall must be crossable by exactly one special mechanism that the hardware controls — not by picking a jump target yourself.

PICTURE. There is exactly one door in the wall, and it is a revolving door bolted to a fixed spot. You can't move the door. You can only spin through it. That door is the trap.

Step 3 — Speaking through the door: the register handoff

WHAT. You can't hand the kernel a function to run (it would never trust a pointer from you). Instead you write a number and some arguments into agreed-upon registers before you knock. On Linux x86-64:

$which service rax = n, up to 6 arguments rdi, rsi, rdx, r10, r8, r9$

$rax = n$ — the syscall number: an index into the kernel's own trusted table (e.g. $n = 1$ is write, $n = 39$ is getpid).
$rdi \dots r9$ — the arguments, in a fixed order both sides agreed on in advance.

WHY a number, not an address? The kernel refuses to jump to an address you supply — that would be a trapdoor into privileged code. A number is safe: the kernel looks it up in its table, so it fully controls where control actually goes. Why these particular registers? Because of a written contract, the ABI — see The C Standard Library and ABI — so neither side has to negotiate at runtime.

PICTURE. Six labelled mailboxes on the user side of the wall. You drop a slip in each, then spin the door.

Step 4 — The atomic crossing: what the hardware does in one shot

WHAT. You execute syscall. In a single, uninterruptible hardware action, the CPU:

$syscall ⟹ [into rcx save PC, into r11 save FLAGS, mode flips CPL \to 0, jump to fixed handler PC \to MSR_{LSTAR}]$

save PC into rcx — the program counter (address of your next user instruction) is stashed, so the kernel can send you back to exactly the right spot later.
save FLAGS into r11 — your condition flags are stashed too, for the same reason.
$CPL \to 0$ — the gate opens; dangerous instructions are now allowed.
$PC \to MSR_{LSTAR}$ — the program counter is loaded from a register the kernel filled at boot with the address of its entry handler. You did not choose this address.

WHY atomic? If the mode flipped before the jump landed, there would be a window where user code runs in kernel mode — a security hole. Doing it as one indivisible step closes that window.

PICTURE. One frame: door mid-spin, four labels flying — PC saved, FLAGS saved, bit flips 3→0, landing point fixed by a pin the kernel hammered in at boot.

Step 5 — Inside the kernel: distrust everything, then dispatch

WHAT. Now $CPL = 0$ . The kernel handler first saves the rest of your registers (so it can give them back untouched), then reads rax, and validates it before using it:

$0 \leq rax < N_{syscalls} ⟹ jump to sys_table [rax]$

$0 \leq rax < N_{syscalls}$ — a bounds check. $N_{syscalls}$ is how many entries the table has (~350 on modern Linux). If your number is out of range, the kernel returns an error instead of trusting it.
$sys_table [rax]$ — the table lookup: index in, trusted kernel function out.

WHY validate? This is the whole point of the wall. A pointer argument (like buf) is also checked to make sure it points into your memory, not the kernel's — otherwise you could read the kernel's secrets by handing it a sneaky address. (That address check leans on Virtual Memory and Page Tables.)

PICTURE. A guard at a desk: rejects out-of-range slips (red), accepts valid ones (teal), then reads the matching row of the table.

Step 6 — Do the work, put the answer back, spin the door home

WHAT. The chosen kernel function does the privileged work (touches the disk, whatever). It writes the result into rax, restores your saved registers, and executes the return-from-trap instruction:

$sysret ⟹ [back to your code PC \leftarrow rcx, restore flags FLAGS \leftarrow r11, gate shuts CPL \to 3]$

$PC \leftarrow rcx$ — reloads the program counter saved in Step 4: you resume on the very next instruction after syscall.
$FLAGS \leftarrow r11$ — your flags come back exactly as they were.
$CPL \to 3$ — the mode bit flips back; the dangerous pile is locked again.

WHY a special return instruction too? Same reason as entry: only a hardware-controlled instruction may lower privilege back and restore PC atomically, so user code can't linger in kernel mode.

PICTURE. The revolving door spins the other way; the answer rides back in the rax mailbox; the bit ticks 0→3.

Step 7 — The degenerate case: the syscall that blocks

WHAT. Not every syscall returns immediately. Consider read on a pipe with no data yet. Steps 1–5 happen as normal, but at the "do the work" moment there is nothing to do yet. The kernel marks your process sleeping and — while still in kernel mode — runs the scheduler to give the CPU to a different process.

WHY does this not freeze the machine? Because you are already inside the kernel ( $CPL = 0$ ), the OS is in control and may voluntarily hand the CPU away. Later, an interrupt (data arrived) wakes you, and only then does Step 6 run and send you back.

PICTURE. A branch in the timeline: normal path returns straight home; blocking path detours through the scheduler (Process Scheduling) — and this detour is a full context switch to another process, which is a different thing from the mode switch. A mode switch stays in your process; a context switch changes which process runs.

The one-picture summary

Everything above, compressed into a single loop: user side, the wall with its one revolving door, the atomic entry (save + flip + fixed jump), the guard's bounds check, the work, and the atomic return. Follow the arrow around once and you have narrated a complete system call.

Recall Feynman: retell the whole walk in plain words

There is one machine but two rulebooks: a full one (kernel) and a limited one (user). A single bit decides which rulebook is active. Your program lives under the limited rulebook, so it cannot touch dangerous hardware. When it needs something dangerous done, it writes a number into a mailbox (which service it wants) plus the arguments, and then knocks with a special knock called a trap. That knock is magic: in one indivisible instant the machine remembers where you were, flips to the full rulebook, and drops you at a fixed spot the owner nailed down long ago — you never get to pick the spot. Inside, a guard checks your number is a real request and that your pointers point at your own stuff, then looks the number up in a trusted list and runs the matching routine. The answer is placed back in the mailbox, your saved position and flags are handed back, and a return-knock flips the bit to the limited rulebook and drops you exactly one step past where you knocked. If the thing you asked for isn't ready yet, the owner — still holding the full rulebook — puts you to sleep and lets somebody else use the machine, waking you when your data shows up. That sleep is the only time a different program runs; every other step keeps you as you, just under a different rulebook.

Recall Quick checks

What flips during a mode switch, and how many bits is it? ::: The CPL / mode bit — a single bit (0 = kernel, 3 = user on x86). Why hand the kernel a number instead of a function address? ::: The kernel refuses to jump to a user-supplied address; a number is an index it looks up in its own trusted table, so it controls where control goes. Which register carries the syscall number, and which carries the return value? ::: rax carries both — the number on the way in, the result on the way out. Where does syscall save your next-instruction address? ::: In rcx (and FLAGS in r11), so sysret can resume you exactly. What extra thing can a blocking syscall trigger that a plain syscall does not? ::: A context switch to a different process (via the scheduler), which is distinct from the mode switch.