std::set_new_handler in C++: Handling Out of Memory

In C++, dynamic memory allocation failures typically result in a std::bad_alloc exception being thrown by operator new. However, real-time, embedded, and mission-critical applications often require deterministic behavior or centralized recovery mechanisms before an exception propagates. This article explains the role of std::set_new_handler, how it integrates into the memory allocation lifecycle, the required responsibilities of a custom handler function, and how to safely manage memory exhaustion.

The Role of std::set_new_handler

Declared in the <new> header, std::set_new_handler registers a user-defined callback function (a std::new_handler) that the default allocation functions invoke when an allocation request cannot be satisfied.

When operator new fails to allocate the requested bytes from the free store, it does not immediately throw an exception. Instead, it checks whether a new-handler function has been registered. If one exists, operator new calls this handler, loops back, and retries the allocation. This loop continues until the allocation succeeds, the handler terminates the process, or the handler throws an exception.

Required Actions of a New-Handler

Because operator new repeatedly invokes the installed handler in an infinite loop until memory becomes available, a custom handler must execute one of the following specific actions to prevent an infinite loop:

1. Make more memory available: The handler can release pre-allocated reserve pools, clear in-memory caches, or free unused subsystems. Upon returning, operator new will attempt the allocation again, which may now succeed.
2. Install a different handler: If the current handler cannot free memory, it can call std::set_new_handler to register a fallback handler capable of taking alternative steps.
3. Uninstall the handler: Passing nullptr to std::set_new_handler removes the current handler. On the subsequent retry, operator new sees no registered handler and throws std::bad_alloc.
4. Throw an exception: The handler can explicitly throw std::bad_alloc or an exception derived from it. operator new will catch nothing and allow the exception to propagate out to the caller.
5. Terminate execution: For systems that cannot recover from out-of-memory states, the handler can log diagnostic information to a low-overhead output stream and safely shut down via std::abort() or std::exit().

Implementation Example

The following pattern demonstrates how to install a custom handler that logs the error, frees emergency reserve memory, and unregisters itself to prevent deadlocks:

#include <iostream>
#include <new>
#include <cstdlib>

char* g_emergency_reserve = nullptr;

void custom_new_handler() {
    // Release reserve memory to allow critical operations or logging to complete
    if (g_emergency_reserve != nullptr) {
        delete[] g_emergency_reserve;
        g_emergency_reserve = nullptr;
        std::cerr << "[Memory Alert] Emergency memory freed. Retrying allocation.\n";
        return; // Returns to operator new to retry allocation
    }

    // If reserve is already exhausted, abort or unregister
    std::cerr << "[Fatal Error] Out of memory. Aborting execution.\n";
    std::set_new_handler(nullptr); // Ensure std::bad_alloc is thrown next, or terminate directly
    std::abort();
}

int main() {
    // Allocate emergency memory reserve
    g_emergency_reserve = new char[1024 * 1024 * 64]; // 64 MB reserve

    // Register the custom handler
    std::set_new_handler(custom_new_handler);

    // Code that triggers massive memory consumption
    // ...

    return 0;
}

Class-Specific Memory Handling

While std::set_new_handler configures a global hook, C++ also supports class-specific handlers. By overloading operator new and providing custom set_new_handler routines for individual classes, developers can tailor out-of-memory responses depending on the operational significance of the object being allocated.