Memory-Mapped File Access with Python mmap

Python's mmap module enables applications to interact with files on disk as if they were contiguous blocks of memory in RAM. By leveraging the operating system's virtual memory subsystem, mmap avoids traditional buffer-copying overhead, provides high-performance random access to large files, and facilitates shared memory between processes. This article explains the underlying mechanics of memory mapping in Python, how the mmap module operates under the hood, and how to implement it effectively for reading and writing data.

How Memory Mapping Works

In standard file I/O operations using read() and write(), data must travel through several layers: the disk, the kernel space buffer cache, and finally the user space buffer allocated by the Python runtime. This double-buffering introduces overhead, especially when dealing with multi-gigabyte files.

The mmap module bypasses this redundant copying by utilizing low-level operating system system calls: mmap() on POSIX/Linux systems and CreateFileMapping combined with MapViewOfFile on Windows. When a file is memory-mapped:

  1. Virtual Address Allocation: The OS assigns a range of virtual memory addresses directly corresponding to the file's bytes.
  2. Demand Paging: The file content is not immediately loaded into physical RAM. Instead, the OS uses page tables to map virtual pages to the file on disk.
  3. Page Faults: When your code accesses a specific byte or slice, the CPU triggers a page fault if that page is not yet in RAM. The OS kernel then loads only that specific 4 KB (or system default) page from the disk.
  4. Automatic Flushing: Modifications made to the memory region are automatically synchronized back to the underlying storage device according to the OS caching policies or explicit flush commands.

Key Capabilities of Python's mmap

Python's mmap object behaves simultaneously like an array-like mutable byte sequence and a file-like object. This duality enables several powerful capabilities:

Reading Files with mmap

To map a file, you must first open it using standard file descriptors, then pass that descriptor to mmap.mmap().

import mmap

with open("large_dataset.bin", "rb") as f:
    # length=0 maps the entire file; access=ACCESS_READ specifies read-only
    with mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_READ) as mm:
        # Read the first 16 bytes
        header = mm[:16]
        
        # Search directly inside the file using regex or find
        position = mm.find(b"TARGET_PATTERN")
        if position != -1:
            print(f"Pattern found at byte offset: {position}")

Because the OS handles caching transparently, searching a 50 GB file with mm.find() uses minimal application memory.

Writing and Modifying Files

To write to a file via memory mapping, open the file in read-write update mode ("r+b") and pass access=mmap.ACCESS_WRITE.

import mmap

with open("data.bin", "r+b") as f:
    with mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_WRITE) as mm:
        # Overwrite bytes in-place
        mm[0:4] = b"TEST"
        
        # Ensure changes are written to the physical storage device immediately
        mm.flush()

The flush() method forces the operating system to write modified memory pages back to the disk, preventing data loss in the event of an abrupt shutdown.

Anonymous Memory Mapping for IPC

Python's mmap module also supports "anonymous" mappings on POSIX platforms, which do not map to an actual file on disk. Passing -1 as the file descriptor creates a block of shared memory in RAM, useful for exchanging data between parent and child processes created via os.fork() without touching persistent storage.

Limitations to Consider