In-Place File Updates with Python mmap.ACCESS_WRITE

Python's mmap module allows applications to map files directly into virtual memory, treating disk-backed storage as a contiguous byte array. By configuring a memory-mapped file with mmap.ACCESS_WRITE, developers grant write-through permissions to the mapping, allowing any modifications made to the in-memory byte buffer to directly update the underlying file on disk. This article explains the underlying mechanism of mmap.ACCESS_WRITE, how the operating system handles synchronized disk writes, and how to execute efficient, in-place data updates in Python without rewriting entire files.

Understanding Memory Mapping and ACCESS_WRITE

Standard file I/O operations require copying data between kernel buffers and user-space memory using system calls like read() and write(). When updating large files, conventional approaches typically involve reading data into memory, modifying it, and rewriting portions or the entirety of the file back to disk.

The mmap module bypasses conventional I/O by utilizing the operating system's virtual memory subsystem. When a file is mapped using mmap.ACCESS_WRITE, the OS associates virtual memory addresses with the file's physical pages on the storage medium. Under this access flag:

How In-Place Updates Work in Practice

To use mmap.ACCESS_WRITE, the target file must first be opened in a read-write binary mode (r+b). Attempting to map a read-only file descriptor with write access raises an OSError or PermissionError.

Here is a practical demonstration of performing an in-place update:

import mmap

# Create a sample binary file
filename = "data.bin"
with open(filename, "wb") as f:
    f.write(b"USER_ID:0000;STATUS:PENDING;END")

# Open for reading and writing in binary mode
with open(filename, "r+b") as f:
    # Map the entire file with write access
    with mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_WRITE) as mm:
        # Locate the target sequence
        target = b"PENDING"
        pos = mm.find(target)
        
        if pos != -1:
            # Perform an in-place update
            replacement = b"SUCCESS"
            mm[pos:pos + len(target)] = replacement
            
            # Explicitly commit changes from memory to disk
            mm.flush()

# Verification
with open(filename, "rb") as f:
    print(f.read())  # Outputs: b'USER_ID:0000;STATUS:SUCCESS;END'

Page Synchronization and Durability

When bytes in an ACCESS_WRITE mapping are updated, the OS marks the corresponding virtual memory page as dirty. The actual disk update occurs asynchronously through the operating system's background page-flushing mechanisms.

To ensure durability before closing the map, Python provides the flush() method. Invoking mm.flush() triggers the underlying msync() (on POSIX) or FlushViewOfFile() (on Windows) system call, forcing the operating system to immediately synchronize modified memory pages to the physical disk.

Critical Constraints