4.2.3Operating Systems

System calls — user mode vs kernel mode, trap mechanism

2,299 words10 min readdifficulty · medium6 backlinks

WHY does this even exist?

  • WHAT is user mode? A restricted CPU state. Cannot run privileged instructions (e.g. HLT, I/O port access, loading the page-table base register).
  • WHAT is kernel mode (a.k.a. supervisor / privileged / ring 0)? Full access to every instruction and all memory.
  • HOW does the hardware know which mode? A bit in a status register (on x86, the CPL = Current Privilege Level, in bits of the CS register; ring 0 = kernel, ring 3 = user).

HOW a system call actually runs (the mechanism, step by step)

Think of it as a derivation: each step exists to solve a problem the previous step created.

  1. Problem: the program wants write(fd, buf, n) but can't touch the disk. → It must enter the kernel.
  2. The C library puts the syscall number into a register (Linux x86-64: number in rax, args in rdi, rsi, rdx, ...).
    • Why a number? The kernel can't trust a raw function pointer from user space; it only accepts an index into its own trusted syscall table.
  3. It executes the trap instruction (syscall).
    • Why a special instruction? A normal call would jump to a user address still in user mode. The trap instruction is the only legal way to flip the mode bit — and it doesn't let user code pick where to land.
  4. Hardware atomically: sets mode = kernel, saves the user PC/flags, and jumps to a fixed handler address the kernel registered at boot.
    • Why fixed? So user code can't redirect the entry into the middle of some privileged routine.
  5. Kernel handler saves remaining registers, reads rax, validates it (in range? pointer valid?), and calls sys_table[rax].
    • Why validate? The whole point of the wall — never trust user-supplied numbers or pointers.
  6. Kernel does the work, puts the return value in rax, restores registers.
  7. Executes return-from-trap (sysret / iret): restores mode = user and PC. Control resumes in your program right after the trap.
Figure — System calls — user mode vs kernel mode, trap mechanism

Traps vs Interrupts vs Exceptions (don't confuse them)


Worked Examples


Common Mistakes (Steel-man + Fix)


Recall Feynman: explain it to a 12-year-old

Imagine a video game where you (a player) can't change the game's rules — only the game master can. When you want something special, like saving your progress, you raise your hand with a special signal. The game master pauses, checks your hand-raise is allowed, saves your game, and lets you keep playing. You never get to touch the rulebook yourself — and that's good, because if every player could edit the rules, the game would break. The "raise hand" signal is a system call, and the moment the game master takes over is kernel mode.


Active Recall

What is the mode bit and where conceptually does it live?
A hardware privilege flag in a CPU status register (x86: CPL in CS); it tells the CPU whether privileged instructions are allowed (kernel) or fault (user).
Define a system call.
A controlled, well-defined entry point letting a user-mode program request a kernel-privileged service via a trap.
Why must the kernel entry point be a FIXED address, not chosen by user code?
So user code cannot jump into the middle of a privileged routine and bypass validation/protection.
Why does a syscall pass a NUMBER instead of a function pointer?
The kernel only trusts an index into its own syscall table; raw pointers from user space are untrusted.
Trap vs Exception vs Interrupt?
Trap = synchronous & intentional (syscall); Exception = synchronous & unintentional (page fault, div0); Interrupt = asynchronous from hardware.
What instruction returns from a syscall and what does it restore?
sysret/iret — restores user mode (mode bit) and the saved user PC/flags.
Difference between a mode switch and a context switch?
Mode switch stays in the same process (user↔kernel); context switch swaps to a different process. A syscall always mode-switches, only sometimes context-switches.
Why are syscalls considered expensive?
Boundary crossing costs pipeline flush, register save/restore, validation, and cache/TLB pollution — beyond the actual work.
What happens on a blocking read with no data?
Kernel marks the process sleeping, runs the scheduler to pick another process, and wakes it via an interrupt when data arrives.
Is printf one system call?
No — formatting happens in user space and output is buffered; write() (the actual syscall) may run once for many printfs.

Connections

Concept Map

cannot run

selects

selects

requests via

entry via

hardware sets

jumps to

indexes

dispatches using

runs

returns to

User mode restricted

Privileged instructions

Mode bit / CPL

Kernel mode full access

App needs kernel service

System call

Trap instruction

Fixed handler address

Syscall number in rax

Syscall table

Privileged service e.g. write

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, tumhara program ek mehmaan hai jo OS ke ghar mein reh raha hai. Wo seedhe disk, memory ka safe, ya hardware ko touch nahi kar sakta — bahut risky hai. To jab usko kuch privileged kaam chahiye (file likhna, memory mangna), wo ek system call karta hai. Iska matlab: number ko register (rax) mein daalo, aur ek special trap instruction (syscall) chala do. Ye instruction hardware se mode bit flip karta hai — user mode se kernel mode — aur ek fixed kernel address pe jump karta hai. Yahi "ghanti bajana" hai.

Important baat: ye normal function call NAHI hai. Normal call user mode mein hi rehta hai. Sirf trap hi privilege wall ko legally cross kar sakta hai, aur user code khud decide nahi kar sakta ki kahan land karna hai — warna koi bhi program kernel ke andar ghuskar sab kuch hack kar leta. Kernel pehle number ko validate karta hai (range mein hai? pointer valid hai?), phir kaam karta hai, result rax mein daalta hai, aur sysret se wapas user mode mein bhej deta hai.

Teen tareeke se kernel mein entry hoti hai, inhe confuse mat karna: Trap (jaan-boojhke, syscall), Exception (galti se, jaise divide-by-zero ya page fault), aur Interrupt (hardware se asynchronous, jaise keyboard). Sirf trap hi tumhara apna request hota hai service ke liye.

Performance ka 80/20 funda: har syscall mehenga hota hai — pipeline flush, register save/restore, cache/TLB pollution. Isiliye printf turant disk pe nahi likhta; wo user space mein buffer karta hai aur baad mein ek hi write syscall maarta hai. Yaad rakho: tezi syscall ke andar ka kaam optimize karne se nahi, balki syscalls ki ginti kam karne se aati hai.

Go deeper — visual, from zero

Test yourself — Operating Systems

Connections