Line-by-Line Memory Profiling in Python

Line-by-line memory profiling in Python allows developers to inspect the exact memory consumption of individual statements within a function. Using the memory_profiler package, you can identify memory leaks, locate heavy allocations, and optimize resource-heavy routines. This article explains the internal mechanics of memory_profiler, how to set it up, how to analyze its output, and key operational considerations when profiling code.

How memory_profiler Works Internally

Under the hood, memory_profiler relies on two primary components: Python's tracing mechanism and process-level memory queries.

1. Execution Tracing: When a function is targeted for profiling, the tool hooks into Python’s internal execution using sys.settrace. This hook fires a callback before and after each line of Python code is executed.
2. Memory Sampling: At each line boundary, the profiler samples the Resident Set Size (RSS) of the current process. It typically uses the psutil library (or queries /proc/$PID/statm on Linux) to obtain the total physical memory mapped to the process from the operating system.
3. Delta Calculation: By comparing the RSS before a line executes to the RSS immediately after, the profiler calculates the exact difference (or "increment") that the execution of that specific line caused.

Basic Setup and Usage

To enable line-by-line profiling, install memory_profiler along with psutil for faster and more accurate sampling:

pip install memory_profiler psutil

Decorate the function you want to inspect with the @profile decorator. You do not need to import @profile into your module; it is dynamically injected into the built-in namespace when running via the profiler CLI.

# example.py

@profile
def allocate_data():
    numbers = [i for i in range(1_000_000)]
    doubled = [x * 2 for x in numbers]
    del numbers
    return doubled

if __name__ == "__main__":
    allocate_data()

Run the script directly through the profiler module:

python -m memory_profiler example.py

Reading the Profiler Output

The profiler outputs a tabular report detailing each line executed within the decorated function:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     3     38.2 MiB     38.2 MiB           1   @profile
     4                                         def allocate_data():
     5     76.5 MiB     38.3 MiB           1       numbers = [i for i in range(1_000_000)]
     6    115.1 MiB     38.6 MiB           1       doubled = [x * 2 for x in numbers]
     7     76.9 MiB    -38.2 MiB           1       del numbers
     8     76.9 MiB      0.0 MiB           1       return doubled

• Line #: The line number in the source file.
• Mem usage: The total memory consumed by the Python process immediately after executing that line.
• Increment: The change in memory consumption relative to the preceding line. Positive values denote new allocations; negative values indicate freed memory.
• Occurrences: How many times the line was executed (particularly useful in loops).
• Line Contents: The actual source code evaluated.

Key Considerations and Caveats

• Runtime Overhead: Because sys.settrace intercepts every execution step, line-by-line profiling introduces significant runtime latency. It should only be used in development or testing environments, never in production.
• Garbage Collection Behavior: Python memory is not always returned to the operating system immediately upon object deletion. CPython uses an internal memory allocator (PyMalloc) for small allocations; memory released by Python objects may be retained within the interpreter pool for reuse rather than reflected as a negative Increment in the OS-level RSS.
• Subprocesses: By default, memory_profiler monitors only the main process. If your functions spawn child processes via multiprocessing, memory usage within those child processes will not be recorded in the line-by-line table unless configured to monitor children explicitly.