Foundations — System calls — user mode vs kernel mode, trap mechanism

This page assumes you know nothing. Before you can follow the parent note System calls — user mode vs kernel mode, trap mechanism, you need a small pile of ideas: what a register is, what a bit is, what "the CPU jumps somewhere" means, and what a privileged instruction is. We build each from zero, in an order where every idea leans only on the ones before it.

0. The CPU, in one picture

Everything below happens inside one chip. Let us draw it before naming anything.

Two things live inside it that we will use constantly: registers and the program counter. Let us earn each.

1. Bit — the smallest thing

Why the topic needs it: the entire "who is in charge?" question — user or kernel — is answered by one single bit inside the CPU. If you do not know what a bit is, the phrase "mode bit" is meaningless.

2. Register — the CPU's hands

Picture a worker with a handful of labelled cups on the desk. Each cup (rax, rdi, rsi, …) holds exactly one number at a time. To do anything, the worker moves numbers into cups, then acts on the cups.

Why the topic needs it: every system call communicates through registers. The parent note says "put the syscall number in rax, args in rdi, rsi, rdx". Those are just named cups. The answer also comes back in a cup (rax). No cups, no system calls.

3. Program Counter (PC) — "where am I on the list?"

Picture a finger pointing at line 7 of the list. Do line 7 → finger slides to line 8. A jump is when the finger suddenly leaps to a different line instead of the next one.

Why the topic needs it: a trap is fundamentally "the finger jumps into the OS's code, then later jumps back to your code". Steps 4 and 7 of the parent's mechanism are entirely about saving the PC (remembering where the finger was) and restoring it (putting the finger back). See also Interrupts and Exceptions, where the same save/restore-PC idea appears.

4. Address & memory — the giant numbered shelf

Why the topic needs it: the whole protection story is about which boxes a program is allowed to read or write. In user mode, most boxes (the ones holding the OS, or another program's passwords) are off-limits. The kernel must validate any address you hand it (parent's Step 5) precisely because you might lie about which box you own. This links forward to Virtual Memory and Page Tables.

5. Instruction — safe vs privileged

Now we can split instructions into two piles, which is the heart of everything.

Think of a kitchen. Chopping your vegetables = ordinary, anyone may do it. Turning on the industrial gas stove or opening the safe = privileged, only the owner may.

Why the topic needs it: "user mode cannot run privileged instructions" is the parent's central rule. Without the two piles, the sentence has no meaning.

6. Mode bit — the switch that decides everything

Combine bit (§1) + register (§2) + privileged (§5):

Look at the figure. The same CPU, the same instructions, one bit flipped. When the bit says "user" and the worker tries a privileged instruction, the silicon slaps its hand: the instruction faults instead of running.

Why the topic needs it: every use of the words user mode, kernel mode, and mode switch in the parent is literally "the state of this one bit / the moment it flips".

7. Jump vs Trap — the guarded doorway

An ordinary jump moves the finger (PC) but does not change the mode bit — you stay weak. So how does a weak program ever become strong? It cannot flip its own mode bit; that would defeat the whole wall.

Picture a single locked door in the wall between the two kitchens. You cannot pick where it leads (the OS bolted it to a spot in advance), and stepping through automatically hands you the owner's apron (kernel mode). That is why the parent insists the entry point is chosen by hardware + kernel, never by your jump target.

Why the topic needs it: this is the mechanism. Steps 3–4 and 7 of the parent are "go through the door" and "come back through it". The relatives of the trap — the exception and the interrupt — enter through other doors and are the subject of Interrupts and Exceptions.

8. Syscall table & syscall number — the numbered menu

Picture a restaurant where you cannot walk into the kitchen. You point at item #39 on the menu. The kitchen decides what "#39" actually cooks. That indirection is the safety.

Why the topic needs it: parent Step 2 ("put the syscall number in rax") and Step 5 ("validate it, call sys_table[rax]") are exactly this menu.

9. ABI — the agreed contract

Picture two people who agreed in advance: "questions go on the blue clipboard, answers come back on the same clipboard." No negotiation needed each time. More in The C Standard Library and ABI.

Why the topic needs it: it explains the parent's "why does the answer come back in rax?" — because the ABI fixed it beforehand.

How these feed the topic

Read it bottom-up: bit + privileged instruction → mode bit → trap → system call, with registers + ABI + syscall number supplying the data that flows through, and address + validation keeping it safe.

Equipment checklist

Cover the right side and test yourself. If any answer surprises you, re-read that section.