Linux System Calls: User Space to Kernel Space

In the Linux operating system, system calls serve as the essential bridge between untrusted user applications and the privileged core of the operating system. Because direct hardware access by user programs would compromise system security and stability, the CPU isolates memory into user space and kernel space. This article explores how system calls safely cross this boundary using hardware-enforced privilege transitions, architecture-specific instructions, and internal kernel dispatch tables.

The Need for Separation: User Space vs. Kernel Space

Modern CPUs utilize privilege rings to enforce boundaries between different types of code. In x86 architecture, these range from Ring 0 (highest privilege) to Ring 3 (lowest privilege). Linux maps these rings into two primary execution environments:

• User Space (Ring 3): Where normal processes, background services, and graphical interfaces execute. Code running here cannot directly access physical hardware, modify page tables, or read memory allocated to other processes.
• Kernel Space (Ring 0): Where the core Linux kernel executes. Code here possesses unrestricted access to hardware, device drivers, interrupt controllers, and physical system memory.

Without a structured mechanism to request services from Ring 0, user-space applications would be incapable of performing essential tasks such as reading files, creating network connections, or allocating new memory blocks.

The Role of the C Standard Library (glibc)

User-space applications rarely invoke system calls directly. Instead, they interact with wrapper functions provided by standard libraries, such as the GNU C Library (glibc).

When a developer calls a POSIX function like open(), read(), or write(), the library function prepares the arguments according to the Application Binary Interface (ABI) of the host architecture. It assigns a unique system call number (such as __NR_write) to a designated CPU register (e.g., RAX on x86_64) and places the function arguments in other specific registers (such as RDI, RSI, and RDX).

Triggering the Transition

Once the registers are configured, the library executes a specialized CPU instruction that triggers a synchronous transition from user mode to kernel mode:

• Modern Systems (syscall/sysret or sysenter/sysexit): Modern x86_64 processors use dedicated, high-speed instructions designed specifically for system calls. The syscall instruction instantly switches the CPU privilege level from Ring 3 to Ring 0, saves the instruction pointer, and points execution to a predefined kernel entry point.
• Legacy Systems (int 0x80): Historically, system calls relied on software interrupts. A program triggered interrupt vector 0x80, forcing the CPU to pause execution, switch to Ring 0, and transfer control to the interrupt handler table.

Regardless of the instruction used, the CPU hardware enforces the privilege elevation and ensures execution can only begin at predetermined entry points specified by the kernel, preventing user code from executing arbitrary kernel addresses.

Kernel Dispatch and Execution

Once the CPU enters Ring 0, the kernel takes control through an entry point typically named entry_SYSCALL_64:

1. State Preservation: The kernel saves the user-space register states onto the process's kernel stack so that execution can resume seamlessly later.
2. Lookup in the System Call Table: The kernel reads the system call number stored in the register. It references the sys_call_table, an array of function pointers mapping system call numbers to their respective kernel implementations (such as sys_read or sys_write).
3. Parameter Verification: Before executing the routine, the kernel strictly validates all incoming pointers and buffer lengths to ensure they originate from valid, accessible user-space memory, preventing privilege escalation vulnerabilities.
4. Routine Execution: The target kernel function executes with full hardware privileges to complete the requested operation.

Returning to User Space

After the kernel function completes its task, it stores the return value (or an error code) in a register (such as RAX). The kernel then restores the saved user-space registers from the kernel stack.

Finally, the kernel executes an exit instruction, such as sysret on x86_64. This instruction lowers the CPU privilege level back to Ring 3, restores the program counter, and hands control back to the standard library wrapper, which then returns the result to the calling application.