Linux Dynamic Kernel Tracing with ftrace

This article provides an architectural overview of how the Linux operating system manages dynamic kernel tracing through the ftrace framework. It covers the end-to-end lifecycle of function tracing, including compile-time instrumentation with compiler flags, the boot-time conversion of call sites into no-operation (NOP) instructions, the runtime dynamic patching mechanism used to hook functions safely across multi-core systems, and the user-space control interface exposed via the tracefs virtual filesystem.

Compile-Time Instrumentation

Dynamic tracing in ftrace begins during the kernel compilation phase. When dynamic ftrace (CONFIG_DYNAMIC_FTRACE) is enabled in the kernel configuration, the build system compiles the kernel code using architecture-specific profiling flags, primarily -pg alongside -mfentry on architectures like x86_64.

The -mfentry compiler option injects a call to a special tracing stub—named __fentry__—at the very beginning of almost every kernel function prologue, before the stack frame is even established. Under standard static tracing, these calls would execute unconditionally, introducing substantial runtime overhead across the entire system.

Boot-Time Conversion to NOPs

To eliminate the performance penalty of unneeded function calls, the kernel neutralizes these trace points during the boot sequence. During the kernel build, a post-processing tool (such as objtool or the historical recordmcount script) scans the compiled object files, records the exact instruction memory addresses of every generated call to __fentry__, and aggregates them into a dedicated kernel section named __mcount_loc.

When the Linux kernel boots, it references this address table and iterates over every instrumented location. It overwrites the multi-byte call instructions with architecture-optimized multi-byte NOP (no-operation) instructions. At this stage, functions execute at near-native speed, paying only the negligible cost of executing a single NOP instruction at the function prologue.

Runtime Binary Patching

Dynamic tracing activates when a tracer (such as the function or function_graph tracer) is enabled by the system administrator. Instead of monitoring all functions globally, the kernel only patches the functions targeted for observation.

Modifying executable code on a live, multi-core operating system presents synchronization hazards; executing partially overwritten instructions on another CPU core causes kernel panics. Linux manages safe live code patching using specialized instruction-updating techniques:

1. Breakpoint Injection: In modern x86 implementations, the kernel replaces the first byte of the target NOP instruction with a temporary software breakpoint (int3).
2. Synchronization: An inter-processor interrupt (IPI) flushes the CPU pipelines across all active cores, ensuring no core is executing a stale decode buffer at that memory location.
3. Payload Replacement: The remainder of the instruction bytes are updated to direct control flow to the ftrace handler (ftrace_caller).
4. Breakpoint Removal: The initial breakpoint byte is rewritten with the first byte of the new call instruction, accompanied by another memory barrier and pipeline sync.

When the trace session ends, the process reverses: the calls to ftrace_caller are hot-patched back into NOPs.

Trampolines and Execution Flow

When a patched function executes, the CPU hits the newly written call instruction and jumps to a specialized trampoline, typically ftrace_caller.

The trampoline performs the following low-level operations:

• Saves the CPU register state to preserve the target function's input arguments.
• Passes two critical parameters to the ftrace engine: the program counter (IP) of the calling function and the parent function's return address.
• Invokes the active ftrace callback functions (for example, logging the event to the per-CPU ring buffer).
• Restores all saved registers.
• Returns execution seamlessly to the original function body.

For the function_graph tracer, the trampoline also hijacks the return path. It pushes the true return address onto an internal ftrace-managed shadow stack and rewrites the function’s actual return address on the CPU stack to point to return_to_handler. When the kernel function finishes, it returns to ftrace first, allowing the engine to calculate and record the function's precise execution duration before jumping back to the actual caller.

The Tracefs Control Interface

Dynamic tracing is managed from user space via the tracefs synthetic filesystem, mounted by default at /sys/kernel/tracing (or /sys/kernel/debug/tracing).

Linux exposes specific control files within this directory to direct the dynamic patching behavior:

• current_tracer: Specifies the tracing engine to activate (e.g., function, function_graph, or nop).
• set_ftrace_filter: Accepts function names, wildcards, or module names. The kernel maps these strings against its internal mcount address tables and dynamically patches only the matching functions from NOP to the trace handler.
• set_ftrace_notrace: Excludes specific functions from being traced, maintaining them as NOPs even if a broader filter would otherwise match them.
• trace: Outputs the recorded execution events pulled sequentially from the lockless per-CPU memory ring buffers.

Through this combination of compiler hooks, boot-time instruction conversion, safe atomic code replacement, and a virtual filesystem interface, Linux achieves dynamic kernel tracing with near-zero overhead when disabled and granular, precise observability when enabled.