Mapping Linux Perf to Python Bytecode Frames

Profiling Python applications with Linux perf presents a unique challenge: perf monitors kernel and native CPU instructions, while Python executes dynamically compiled bytecode within a virtual machine. This article explains how Linux perf tracepoints and sampling interrupts can be bridged to dynamic CPython execution states. It details how to leverage native perf trampolines introduced in CPython 3.12, User Statically Defined Tracing (USDT) probes, and eBPF-driven frame inspection to associate low-level events directly with Python functions, filenames, and bytecode offsets.

The Challenge: The Native-to-Bytecode Gap

When running standard perf record on a CPython binary, stack unwinding only resolves native C symbols:

_PyEval_EvalFrameDefault (in libpython3.x.so)
  └── PyObject_Vectorcall
        └── ...

Because CPython executes Python bytecode inside an evaluation loop (historically eval_frame or _PyEval_EvalFrameDefault), the native CPU instruction pointer (RIP/PC) points to the machine code of the interpreter rather than the Python source code. To map kernel tracepoints (e.g., sched:sched_switch, page faults, system calls) to Python bytecode, you must map the active PyFrameObject (or _PyInterpreterFrame in Python 3.11+) to the hardware instruction pointer at the moment the tracepoint fires.

Method 1: Python 3.12+ Native Perf Support (Perf Maps and Trampolines)

CPython 3.12 introduced native support for Linux perf via runtime trampoline generation. Instead of calling the evaluation loop generically, Python generates tiny JIT-like native trampolines for each Python code object.

1. How It Works

When a Python function is executed:

1. CPython writes machine code stubs (trampolines) that call into the frame evaluator.
2. The runtime registers these addresses and writes entries to /tmp/perf-<PID>.map or emits a binary jitdump file.
3. Linux perf parses /tmp/perf-<PID>.map during symbol resolution to map instruction addresses directly to Python function names, files, and line numbers.

2. Activation

Enable trampolines via an environment variable or command-line flag:

# Via environment variable
export PYTHONPERFSUPPORT=1
perf record -F 999 -g -- python your_script.py

# Or via the CLI switch
perf record -F 999 -g -- python -X perf your_script.py

3. Analyzing Tracepoints

To trace a specific kernel event (e.g., system calls) correlated with Python execution:

perf record -e raw_syscalls:sys_enter -g -- python -X perf your_script.py
perf report --stdio

The call tree will display the Python function name and source file interleaved with native C library calls and kernel functions.

Method 2: USDT Tracepoints in CPython

If CPython is compiled with DTrace/USDT support (--with-dtrace), the binary contains static tracepoints compiled directly into the interpreter.

1. Available USDT Probes

Common probes embedded in the Python runtime include:

• python:function__entry: Fires when entering a Python frame. Arguments provide the filename, function name, and line number.
• python:function__return: Fires when exiting a Python frame.
• python:line: Fires on each bytecode line execution (disabled by default due to high overhead).

2. Registering Probes with Linux Perf

You can inspect and register these probes using perf:

# Check for USDT markers in the python binary
perf list sdt

# Add a USDT marker to perf dynamic tracing
perf probe -x /usr/bin/python3.11 python:function__entry

Once registered, tracepoint events can be recorded alongside hardware performance counters or kernel scheduler events to reconstruct frame lifecycles.

Method 3: Dynamic Bytecode Introspection via eBPF

For environments running Python versions prior to 3.12 without trampolines, eBPF (extended Berkeley Packet Filter) allows out-of-band unwinding of the CPython frame structures.

1. Reading CPython Thread State

In CPython, each OS thread maintains a pointer to a PyThreadState struct, which contains a pointer to the active frame:

• Python 3.10 and earlier: PyThreadState -> frame (PyFrameObject*)
• Python 3.11+: PyThreadState -> current_frame (_PyInterpreterFrame*)

Using an eBPF program attached to a tracepoint (e.g., tracepoint/sched/sched_switch), you can:

1. Locate the base address of libpython in user space.
2. Read the thread-local _PyThreadState_Current symbol using bpf_probe_read_user().
3. Traverse the frame linked list back to find:
   • co_filename: frame->f_code->co_filename
   • co_name: frame->f_code->co_name
   • f_lasti: Current bytecode instruction index inside f_code->co_code.

2. Resolving Bytecode Offsets

The f_lasti attribute represents the exact offset within the bytecode payload. By extracting the bytecode mapping table (co_linetable or co_lnotab), user-space consumers of the eBPF perf buffer match the exact f_lasti integer to the Python source line.

Comparison of Methods

Recommended Workflow

1. Use Python 3.12+ with -X perf as the default mechanism for profiling CPU hotspots, page faults, and context switches alongside Python frames.
2. Use eBPF (or tools like py-spy) when you need non-invasive observation in production environments without restarting the target process or modifying runtime flags.