How Future.add_done_callback Works in Python

The concurrent.futures.Future.add_done_callback() method provides an event-driven mechanism to handle task completion without blocking program execution. Instead of repeatedly polling Future.done() or calling Future.result(), developers can attach callable functions that automatically execute as soon as a future enters a finished state. This article explains the internal mechanics of add_done_callback(), including registration, execution contexts across thread and process pools, argument passing, and exception handling.

Registration and Immediate Execution

When you pass a callable to add_done_callback(fn), the Future instance first acquires its internal reentrant lock (threading.RLock). It evaluates its current state:

1. Pending/Running: If the task is still executing or waiting to run, the function reference is appended to an internal list called _done_callbacks.
2. Already Completed or Cancelled: If the future has already finished execution or was cancelled before the callback was attached, the callback is executed immediately within the calling thread that invoked add_done_callback().

Execution Thread and Context

The execution environment of the callback depends on which executor type is in use and when the future finishes:

• ThreadPoolExecutor: When a worker thread completes the assigned task, it marks the future as finished, acquires the lock, and immediately runs all registered callbacks sequentially in the same worker thread before returning to the pool.
• ProcessPoolExecutor: Worker processes cannot directly run callbacks in the main process. Instead, an internal background management thread (the queue manager thread) in the host process monitors task completion IPC queues. Once the result is deserialized in the main process, this manager thread marks the future as done and executes the callbacks.
• Immediate Fallback: If the future is already marked done prior to registering the callback, the callback executes synchronously on whichever thread invoked add_done_callback().

Signature and Invocation Order

Callbacks attached to a Future must accept exactly one parameter: the Future instance itself.

def my_callback(future):
    try:
        result = future.result()
        print(f"Task succeeded: {result}")
    except Exception as exc:
        print(f"Task generated an exception: {exc}")

If multiple callbacks are attached to a single future, they are invoked in the exact order they were registered (first-in, first-out). Both normal completion, raised exceptions, and explicit cancellations trigger the callbacks, as all three transitions represent terminal states for a future.

Error Handling Inside Callbacks

If a callback function raises an unhandled exception during its execution, it does not stop the execution of subsequent registered callbacks, nor does it crash the worker thread or alter the state of the future. The concurrent.futures implementation catches any Exception raised by a callback and logs it directly to the root logger using logging.exception(). Because callback failures fail silently unless logs are monitored, any critical error-handling logic must be explicitly wrapped in try...except blocks within the callback itself.