Python Multiprocessing: Fork vs Spawn vs Forkserver

Python's multiprocessing module allows developers to bypass the Global Interpreter Lock (GIL) by running tasks across multiple operating system processes. To create these worker processes, Python provides three distinct start methods: fork, spawn, and forkserver. This article breaks down how each method functions, their performance and safety trade-offs, operating system availability, and how to select the right approach for your concurrent applications.

1. The fork Method

The fork method uses the standard POSIX system call os.fork(). When a new worker process is created, the operating system clones the parent process entirely, copying its virtual memory space using copy-on-write (COW).

• How it works: The child process starts with an identical copy of the parent process's state, including loaded modules, global variables, and memory allocations, without re-executing the Python script or re-importing libraries.
• Speed: Extremely fast. Because the OS merely duplicates page tables rather than reloading the Python interpreter, initialization overhead is minimal.
• Drawbacks and Risks:
  • Thread Safety Issues: If the parent process contains active threads, fork only copies the thread that called fork(). The child inherits the memory state of other threads, including any locks or mutexes that were held at the moment of the fork. This frequently causes deadlocks.
  • Platform Support: Available only on Unix-like systems (Linux, macOS). It is not supported on Windows. Due to safety issues, macOS deprecated fork as the default in Python 3.8.

2. The spawn Method

The spawn method launches a completely fresh Python interpreter process from scratch.

• How it works: The operating system creates a new process, initializes a new Python interpreter, and executes the code necessary to import the target function. Only resources explicitly passed as arguments to the target function are transferred, serialized via pickle.
• Speed: The slowest of the three methods. Launching a new interpreter and re-importing all modules in every worker introduces noticeable latency and CPU overhead during startup.
• Advantages:
  • Safety: It guarantees a pristine process state. No unwanted file descriptors, thread-held locks, or corrupted memory states are inherited from the parent.
  • Portability: Supported across all major operating systems. It is the default on Windows and macOS.
• Requirements: All arguments passed to child processes must be serializable via pickle. Code execution must also be guarded by an if __name__ == '__main__': block to prevent infinite recursion during process initialization.

3. The forkserver Method

The forkserver method offers a hybrid approach designed to combine the safety of spawn with the creation speed of fork.

• How it works: When the program requests the forkserver method, a dedicated single-threaded server process is spawned. From that point forward, whenever a new worker process is required, the parent asks this clean server process to fork() itself.
• Speed: Faster than spawn because new workers are forked from an existing process, but slightly slower than pure fork due to the inter-process communication with the server.
• Advantages: Because the fork server process remains single-threaded, workers never inherit unreleased thread locks or corrupted states from the main application process.
• Platform Support: Available only on Unix platforms that support passing file descriptors over Unix domain sockets.

Quick Comparison

How to Configure the Start Method

You can set the start method globally at the start of your program using multiprocessing.set_start_method(). This call should be placed inside the if __name__ == '__main__': block and can only be set once per run.

import multiprocessing

if __name__ == '__main__':
    # Options: 'fork', 'spawn', or 'forkserver'
    multiprocessing.set_start_method('spawn')
    
    # Alternatively, create an isolated context:
    # ctx = multiprocessing.get_context('forkserver')
    # p = ctx.Process(target=worker_func)

Use spawn when writing cross-platform code or working with multi-threaded libraries like NumPy, OpenCV, or PyTorch. Use forkserver on Linux when you need high-throughput process creation without sacrificing thread safety. Use fork strictly in simple, single-threaded Linux applications where process creation speed is paramount.