System calls — user mode vs kernel mode, trap mechanism: Question bank

This is the conceptual companion to the mechanism note (Hinglish version here). Prerequisite ideas linked inline: Interrupts and Exceptions, Context Switching, Process Scheduling, Virtual Memory and Page Tables, CPU Privilege Rings (x86), The C Standard Library and ABI, io_uring and Batched I/O.

True or false — justify

A system call is a controlled request from a restricted program (user mode) into the trusted OS (kernel mode), crossing the privilege wall through a hardware trap. Keep that picture as you judge each claim.

A user-mode program can run any CPU instruction, it just chooses not to run dangerous ones.

False — the hardware itself faults on privileged instructions in user mode. It's silicon-enforced protection, not politeness. See CPU Privilege Rings (x86).

The syscall instruction lets the program pick which kernel address to jump into.

False — user code only supplies a number in rax; the hardware jumps to a fixed, boot-registered entry point. Letting user code choose the target would defeat the wall.

A system call is basically a function call into the OS.

False — a normal call stays in user mode and jumps to a user address. A syscall atomically flips the mode bit and enters a hardware-chosen handler; the mode change is the essential difference.

Kernel mode runs on a separate, more powerful processor.

False — it's the same CPU with one status bit flipped. Same registers, same instruction set; user mode merely forbids the privileged subset.

Every printf in your C program triggers a system call.

False — printf formats and buffers in user-space libc memory, then calls write only occasionally, possibly once for many printfs.

A mode switch and a context switch are the same event.

False — a mode switch stays inside the same process (you, now in the kernel); a context switch swaps to a different process. A syscall always does the first, only sometimes the second. See Context Switching.

Doing more system calls makes a program faster because kernel code is highly optimized.

False — the boundary crossing itself costs hundreds of cycles (pipeline flush, cache/TLB pollution). Fewer, bigger syscalls usually win, which is why io_uring and Batched I/O exists.

A page fault is a kind of system call.

False — a page fault is an exception (unintentional, caused by an instruction going wrong), not a trap you deliberately requested. Both enter the kernel, but only the trap is a service request. See Interrupts and Exceptions.

While your process is blocked inside a read, the whole machine is frozen.

False — inside the kernel the OS controls the CPU and can hand it to another process via the scheduler. Blocking I/O yields, it does not freeze.

The return value of a syscall comes back in rax because the kernel decided so at runtime.

False — the return register is fixed by the ABI (calling convention) ahead of time, so both sides agree without negotiating. See The C Standard Library and ABI.

Spot the error

Each statement below is almost right. Name the specific broken piece.

"The syscall number is a function pointer the kernel calls directly."

The error is "pointer" — it's an index into the kernel's own trusted syscall table. A raw pointer from user space would let the program run arbitrary kernel code.

"After syscall, the CPU jumps to wherever the program's return address points."

Wrong direction — syscall jumps to the kernel's fixed entry handler; the user return address is what gets saved so sysret can restore it later.

"The kernel trusts the pointer in rsi because it came through a legitimate trap."

Passing through a legal trap does not validate arguments. The kernel must still check every user pointer (in range, mapped, permitted) — the trap only changes mode, not trust.

"int 0x80, syscall, and svc are three different services."

They are three different trap instructions (x86 legacy, x86-64, ARM) that all achieve the same thing: flip to kernel mode and enter the registered handler. Not different services.

"A context switch is required for every system call."

Only a mode switch is required. A context switch happens only if the syscall blocks or the scheduler decides to preempt — most syscalls return to the same process.

"Interrupts, exceptions, and traps are just three names for the same mechanism."

They differ by cause: interrupt = asynchronous hardware, exception = synchronous unintended fault, trap = synchronous intentional request. They share the kernel-entry destination, not the cause.

Why questions

Why must the trap instruction be the only legal way to raise the privilege level?

If any ordinary jump could raise privilege, malicious code would jump into the middle of a privileged routine and skip the checks. Restricting it to a hardware trap forces entry through the vetted handler.

Why does the kernel pass a number instead of accepting a function pointer from user space?

A number is only an index into the kernel's own table, so user code can never point at arbitrary code. It's the difference between "request menu item 39" and "run this address I chose."

Why is the kernel entry address fixed at boot rather than supplied per call?

A fixed, pre-registered address means user code cannot redirect entry into the middle of a privileged routine, bypassing validation. Fixedness is what makes the wall trustworthy.

Why is a system call considered "expensive" even when the work inside is tiny (like getpid)?

The cost is dominated by the crossing — pipeline flush, register save/restore, cache/TLB pollution — not the work. That fixed overhead is paid every time regardless of how small the job is.

Why does blocking inside a syscall help overall system throughput instead of wasting time?

Because the blocked process voluntarily gives the CPU to the scheduler, which runs someone else. Idle-waiting one process becomes useful work for another. See Context Switching.

Why validate user pointers if the program "obviously" passed a valid buffer?

Because user code may be buggy or malicious and could pass a pointer into kernel memory or another process's pages. The kernel checks against the process's page tables before ever dereferencing.

Why does batching syscalls (e.g. writev, io_uring) speed programs up so dramatically?

It amortizes the fixed per-crossing overhead across many operations, turning N expensive boundary crossings into one. See io_uring and Batched I/O.

Edge cases

What happens if a program executes a privileged instruction (like HLT) while in user mode?

The CPU raises an exception (a fault), not the instruction — control jumps to the kernel's fault handler, which typically kills the process. The privileged op never runs.

What if rax holds a syscall number that doesn't exist in the table?

The kernel's validation step catches the out-of-range index and returns an error (e.g. -ENOSYS) instead of indexing past the table. This is exactly why the range check exists.

Can a system call return without ever running kernel service code?

Yes — if validation fails (bad number, bad pointer), the kernel returns an error immediately. The trap and mode switch still happened, but no service was performed.

Does returning from a trap (sysret/iret) always resume the same instruction that trapped?

No — it resumes at the instruction after the trap, because the trap was intentional and completed. (Contrast a page-fault exception, which typically re-runs the faulting instruction after fixing the mapping.) See Interrupts and Exceptions.

If an interrupt (timer, keyboard) fires while your process is mid-syscall in the kernel, is that a nested system call?

No — it's an asynchronous interrupt, not a trap. It interrupts kernel execution, runs its handler, then resumes your syscall. It was never a service request from your program.

What is the mode bit's value at the exact instant the hardware jumps to the trap handler — user or kernel?

Kernel — the flip to kernel mode and the jump happen atomically as one hardware action, so there is no window where the handler runs with user privileges.

Does a syscall that blocks and later resumes count as one mode switch or two?

Potentially two boundary crossings' worth of state work — you enter (mode switch), block (context switch away), and eventually the kernel restores your context and returns (mode switch back). The single logical syscall spans multiple transitions. See Context Switching.