System calls — user mode vs kernel mode, trap mechanism: Worked examples

You have met the parent idea: a program rings a bell (a trap), the kernel checks the request, does it, and returns. That story had exactly one happy ending. Real programs hit many endings — good returns, error returns, blocked calls, calls that never even reach the kernel, calls that trigger a context switch, calls whose numbers are garbage.

This page walks every one of those endings. First we lay out a matrix of all the case classes. Then every example below is stamped with the matrix cell it covers, so by the end no scenario is left dark.

Before anything else, three tiny vocabulary anchors so nothing appears un-earned:

The scenario matrix

Every syscall outcome falls into one of these cells. The examples that follow are labelled by cell ID.

Cell	Case class	What varies	Example
A	Success, no I/O wait	`rax` ≥ 0, returns instantly	Ex 1 (`getpid`)
B	Success, returns a count	`rax` = bytes moved (could be less than asked)	Ex 2 (`read` partial)
C	Failure (negative `rax`)	`rax` < 0 → an errno	Ex 3 (`open` on missing file)
D	Degenerate input: zero-size request	asked for 0 bytes	Ex 4 (`read(fd, buf, 0)`)
E	Blocking → context switch	syscall sleeps, scheduler runs	Ex 5 (`read` empty pipe)
F	"Not actually a syscall"	request served in user space	Ex 6 (`printf` buffering)
G	Invalid / out-of-range syscall number	kernel rejects the index	Ex 7 (`rax = 999999`)
H	Bad pointer argument	kernel must validate memory	Ex 8 (`read` into NULL)
I	Real-world word problem + cost	counting the price of many crossings	Ex 9 (log writer)
J	Exam twist: trap vs interrupt vs exception	classify who entered the kernel	Ex 10
K	Interrupted by a signal	blocked syscall returns `-EINTR`, maybe restarts	Ex 11
L	Non-blocking, nothing ready	`-EAGAIN` / `-EWOULDBLOCK` instead of sleeping	Ex 12

We will show, for the sign of rax, all three regions — negative (error), zero (degenerate/EOF), positive (success) — because that single number decides your program's fate.

Figure — System calls — user mode vs kernel mode, trap mechanism

Example 1 — Cell A: instant success

Worked example

getpid() — the simplest possible crossing

mov rax, 39      ; syscall number for getpid
syscall          ; TRAP -> kernel mode
; rax now holds the PID

Suppose this process has PID 4021. What is in rax after syscall?

Forecast: guess before reading — will rax be 39, 4021, or 0?

mov rax, 39 puts the syscall number into the request cup. Why this step? The kernel trusts only an index into its own table, never a raw address. 39 is that index for getpid.
syscall flips the mode bit and jumps to the kernel's fixed entry. Why this step? It is the only legal instruction that crosses the privilege wall without letting user code pick the landing spot.
The kernel reads the process's own PID field and writes it into rax, then returns. Why this step? The ABI fixes rax as the return cup, so both sides agree with no negotiation.

Answer: rax = 4021.

Verify: the number 39 (the input) is overwritten by the output; a positive rax (4021 > 0) means success — matches Cell A. getpid never fails, so we should never see a negative here. This lands in the mint region of Figure 1. ✓

Example 2 — Cell B: success that returns less than you asked

Worked example

read(fd, buf, 100) from a file with only 30 bytes left You ask for 100 bytes. The file has 30 bytes before end-of-file.

Forecast: will rax come back as 100, 30, or an error?

Put args in cups: rax=0 (read's number), rdi=fd, rsi=buf, rdx=100. Why this step? The count 100 is a maximum, a "give me up to this many", not a demand.
syscall → kernel copies the 30 available bytes into your buffer. Why this step? The kernel copies what exists; it will not invent 70 bytes that aren't there.
Kernel returns rax = 30. Why this step? The return value is the actual count moved, so your program knows how much it really got.

Answer: rax = 30. A short read.

Verify: 0 < 30 < 100. This is the classic bug source: never assume rax == 100. Sum of two reads on this file would be 30 + 0 = 30 bytes total, then the next read returns 0 (EOF, see Cell D reasoning). ✓

Example 3 — Cell C: the failure region (negative `rax`)

Worked example

open("/no/such/file", O_RDONLY) The file does not exist. On Linux the error "no such file or directory" is ENOENT, numeric value 2.

Forecast: positive number? zero? or something negative?

rax = 2 (open's syscall number), rdi = pointer to the path string, rsi = flags. Why this step? open needs the path and mode; both go through the standard arg cups.
syscall → kernel walks the directory tree, finds nothing. Why this step? The kernel must resolve the path itself; user space cannot be trusted to say "yes it exists".
Kernel returns rax = -2 (that is, -ENOENT). Why this step? The errno convention encodes failure as a negative return, magnitude = error code.
The C library sees the negative value, stores 2 into the global errno, and returns -1 to your C code. Why this step? C programmers check if (fd == -1) then read errno; the wrapper translates the raw negative into that idiom.

Answer: raw kernel rax = -2; C-level return -1 with errno = 2.

Verify: sign is negative → Cell C confirmed. And -(-2) = 2 = ENOENT, matching the errno stored. This is the leftmost coral landmark in Figure 1. ✓

Example 4 — Cell D: degenerate zero-size request

Example 5 — Cell E: blocking, and the context switch it triggers

Worked example

read on an empty pipe (no writer has sent data yet) Forecast: does your program spin? freeze the whole machine? or does someone else get the CPU?

read traps into the kernel with a nonzero count. Why this step? Same crossing as always; the difference is what the kernel finds.
Kernel checks the pipe buffer — empty. It marks your process sleeping and does not return. Why this step? There is nothing to copy, so returning now would be wrong; blocking waits for data.
Kernel invokes the scheduler, which picks another ready process, and performs a context switch. Why this step? The CPU should not idle; while you sleep, another program runs. This is why blocking I/O doesn't freeze the machine.
Later, a writer sends bytes → hardware/software delivers data → kernel marks you runnable, eventually a switch brings you back, and now read completes returning rax = bytes delivered. Why this step? Only once data exists can the copy happen and the syscall finish.

Answer: the single read call spans a mode switch and at least two context switches, finally returning a positive rax.

Verify: distinct concepts confirmed — mode switch stays in your process; the context switch swaps to another process. A blocking syscall does both; getpid (Ex 1) does only the mode switch. ✓

Example 6 — Cell F: the call that isn't a syscall (yet)

Worked example

printf("Score: %d\n", 42) — how many traps? Forecast: 1 syscall per printf? 2? 0?

printf formats "Score: 42\n" entirely in user space memory. Why this step? Formatting numbers into text needs no privilege, so no crossing is required.
The result is copied into the C library's private buffer (stdio buffer), still user space. Why this step? Batching. Each write crossing is expensive; collecting output delays the price.
Only when the buffer fills, or a newline flushes a line-buffered terminal, or the program exits, does the library issue one write(1, buf, n) — the sole actual trap. Why this step? One trap can flush many printfs, turning N expensive crossings into 1.

Answer: a single printf may cause zero syscalls right now (data sits in the buffer) and share one later write with hundreds of siblings.

Verify: if 1000 printfs each formatting 10 bytes flush in one write of a 4096-byte buffer, the byte count moved is 1000 × 10 = 10000 bytes, needing ⌈10000 / 4096⌉ = 3 write syscalls, not 1000. Crossings dropped by over 300×. ✓

Example 7 — Cell G: garbage syscall number

Worked example

mov rax, 999999 then syscall There is no syscall numbered 999999.

Forecast: does the kernel crash? run random code at index 999999? or reject it?

rax = 999999, then syscall. Why this step? We deliberately hand the kernel an out-of-range index.
Kernel's dispatcher checks rax against the table size before using it as an index. Why this step? Using an unchecked index would read past the table into arbitrary memory — the exact attack the privilege wall exists to stop.
Out of range → kernel returns rax = -38 (that is -ENOSYS, "function not implemented", value 38). Why this step? A clean, negative error instead of undefined behaviour.

Answer: rax = -38, no crash, no code executed at 999999.

Verify: negative → Cell C-style failure, magnitude 38 = ENOSYS. -(-38) = 38. The bounds check is why a bad number can't hijack the kernel. ✓

Example 8 — Cell H: bad pointer argument

Worked example

read(fd, NULL, 100) — buffer pointer is address 0 Forecast: does the kernel write to address 0 and crash the machine? or refuse?

rax=0, rdi=fd, rsi=0 (NULL), rdx=100. Why this step? We feed the kernel a pointer that user space is not allowed to touch.
Before copying, the kernel validates that the destination range [0, 100) lies in your legal address space. Why this step? The kernel must never blindly dereference a user pointer — see page tables for how it checks. Trusting the pointer would let a program trick the kernel into writing anywhere.
Address 0 is not a valid user page → kernel aborts the copy and returns rax = -14 (-EFAULT, "bad address", value 14). Why this step? Report a clean error rather than corrupt memory.

Answer: rax = -14, no corruption.

Verify: negative → failure; -(-14) = 14 = EFAULT. Note this is a synchronous, unintentional-looking rejection but it is inside an intentional trap — still Cell H, distinct from a hardware page-fault exception in Ex 10. ✓

Example 9 — Cell I: word problem, counting the real cost

Two symbols appear below; earn them first:

Worked example A logger writes 1 line (80 bytes) per event, 50000 events, using the cost model

Assume one write crossing costs a fixed T_cross = 1000 ns of overhead (mode switch + save/restore + cache pollution), and the copy of 80 bytes costs T_work = 20 ns. Compare unbuffered (one write per line) vs buffered (fill a 4096-byte buffer, then flush).

Forecast: guess the speedup factor — 2×? 10×? 50×?

Unbuffered: 50000 lines → 50000 write syscalls. Time $= 50000 \times (T_{cross} + T_{work}) = 50000 \times (1000 + 20)$ ns. Why this step? Every line pays the full crossing price.
Compute: $50000 \times 1020 = 51, 000, 000$ ns $= 51$ ms. Why this step? We must reduce the formula to a single concrete number of nanoseconds so we can compare it against the buffered case; a formula alone can't be judged "fast" or "slow", only a number can. Dividing by $1 0^{6}$ converts nanoseconds to milliseconds for a human-readable feel.
Buffered: lines per flush $= ⌊ 4096/80 ⌋ = 51$ lines. Number of flushes $= ⌈ 50000/51 ⌉ = 981$ write syscalls. Why this step? One crossing now empties 51 lines' worth of buffer; the floor/ceiling appear because partial lines and partial buffers can't exist.
Time $\approx 981 \times T_{cross} + 50000 \times T_{work} = 981 \times 1000 + 50000 \times 20$ ns $= 981, 000 + 1, 000, 000 = 1, 981, 000$ ns $\approx 1.98$ ms. Why this step? The 80-byte copies still all happen; only the crossings collapse.

Answer: unbuffered ≈ 51 ms, buffered ≈ 1.98 ms → about 25.7× faster.

Verify: speedup $= 51, 000, 000/1, 981, 000 \approx 25.7$ . The win comes almost entirely from cutting crossings 50000 → 981, confirming the 80/20 insight: reduce syscall count, not the work inside. For even more, io_uring batches submissions so many operations share one crossing. ✓

Example 10 — Cell J: exam twist, classify the entry

Worked example For each event, name the entry type: trap, exception (fault), or interrupt

Reference Interrupts and Exceptions for the definitions.

Event	Synchronous?	Intentional?	Type
Program runs `syscall` to call `write`	Yes	Yes	Trap
Program divides by zero	Yes	No	Exception (fault)
Program touches an unmapped page	Yes	No	Exception (page fault)
Keyboard key pressed while program computes	No	No	Interrupt
Timer fires to preempt the process	No	No	Interrupt

Forecast: before reading the table, decide which two columns you'd need to tell all three apart.

Why the two columns decide it?

Synchronous = caused by the current instruction (trap and exception) vs asynchronous = from outside hardware (interrupt).
Among the synchronous ones, intentional = trap (you asked for a service), unintentional = exception (your instruction went wrong).

Verify: all three end in kernel code, but only the trap was requested by the program to obtain a service. The timer interrupt is what forces the preemptive context switch even when a program never calls a syscall. ✓

Example 11 — Cell K: a blocked syscall interrupted by a signal

Worked example

read is sleeping on an empty pipe when a signal (e.g. SIGCHLD) arrives A signal is the kernel's way of tapping your process on the shoulder ("a child exited", "you pressed Ctrl-C"). What happens to the read that was patiently asleep?

Forecast: does read finish with 0 bytes? wait forever? or fail with an error you must handle?

read traps in, finds the pipe empty, and sleeps — exactly as in Ex 5. Why this step? Same blocking path; the twist is what interrupts the sleep.
A signal becomes deliverable to your process. The kernel must run your signal handler now, so it cannot leave read sleeping. Why this step? Signal delivery is urgent and runs in user mode; the kernel has to unwind out of the blocked syscall to get there.
The kernel aborts the sleep and returns rax = -4 (-EINTR, "interrupted system call", value 4). Your signal handler runs. Why this step? -EINTR is the honest report: "I did not finish; a signal cut me short." No data was moved.
Your code must decide: retry the read (restart semantics) or give up. Many programs wrap the call in a loop while ((n = read(...)) < 0 && errno == EINTR);. Why this step? Because -EINTR is not a real I/O failure — the data may still be coming — so the standard fix is to simply call read again.

Answer: rax = -4 (-EINTR), zero bytes moved; correct code retries the call.

Verify: negative → coral failure region of Figure 1, magnitude 4 = EINTR, -(-4) = 4. Distinct from EOF (0): EINTR means "ask me again", EOF means "there is nothing more, ever". ✓

Example 12 — Cell L: non-blocking I/O with nothing ready

Worked example

read on a pipe opened non-blocking (O_NONBLOCK), pipe currently empty You explicitly told the kernel "never put me to sleep — if there's no data, just say so." Now the pipe is empty.

Forecast: does read block like Ex 5? return 0? or return an error?

The file was opened with the O_NONBLOCK flag, so its "don't sleep" mode bit is set. Why this step? This flag is exactly what separates Cell L from the blocking Cell E — same read, opposite policy on emptiness.
read traps in, kernel finds the pipe empty. Why this step? Same discovery as Ex 5; the difference is the response to emptiness.
Instead of sleeping, the kernel returns immediately with rax = -11 (-EAGAIN, also spelled EWOULDBLOCK, value 11). Why this step? The flag forbids blocking, so the kernel's honest answer is "nothing right now — try again later."
Your program does not wait; it goes off to do other work (or asks an event system like epoll to tell it when data arrives). Why this step? Non-blocking I/O exists so one thread can juggle many connections without ever freezing on any single one.

Answer: rax = -11 (-EAGAIN / -EWOULDBLOCK), no bytes, no sleep, no context switch.

Verify: negative → coral region, magnitude 11 = EAGAIN, -(-11) = 11. Crucially this is the only way "no data" appears as a negative rather than a blocking sleep — contrast Ex 5 (sleeps) and Ex 2's EOF (returns 0). io_uring is a modern way to avoid spinning on repeated -EAGAIN. ✓

Recall Quick self-test on the sign of

rax rax after a syscall is -2. Success or failure, and what does it mean? ::: Failure — negative means -errno; -2 = -ENOENT, "no such file or directory". You called read(fd, buf, 100) and got rax = 30. Bug or normal? ::: Normal — a short read; you got 30 of up to 100 bytes. Loop for the rest. You called read(fd, buf, 100) and got rax = 0. Meaning? ::: End-of-file (because you asked for more than 0). No more data will come. You called read(fd, buf, 0) and got rax = 0. Meaning? ::: You asked for nothing and got nothing — success, not EOF. A single read triggered a context switch. What happened? ::: It blocked (no data); the scheduler ran another process while you slept. read returned -4 (EINTR). What do you do? ::: A signal interrupted the blocked call; retry the read (it never really finished). A non-blocking read returned -11 (EAGAIN). Meaning? ::: No data is ready right now; don't block — wait for readiness and try again.