Accelerating AI Recovery with Multi-Tiered NVMe Checkpointing

Artificial intelligence training runs require massive clusters of GPUs operating continuously for weeks, making hardware failures an inevitability rather than a mere possibility. When a failure occurs, the system must roll back to its last saved state, or checkpoint, resulting in lost compute time and wasted financial resources. Asynchronous multi-tiered checkpointing to local NVMe drives solves this issue by moving the initial data dump off slow network storage and onto ultra-fast local solid-state drives. By combining local hardware speed with background processing, this method allows AI clusters to save their progress frequently with near-zero pauses in training, drastically reducing the amount of work lost when a crash happens.

The Bottleneck of Traditional Checkpointing

During large-scale AI training, a model’s state consists of billions or trillions of parameters, gradients, and optimizer states, often totaling terabytes of data. Historically, saving a checkpoint meant pausing the entire training run while every GPU wrote its current state across a network to a centralized Parallel File System (PFS).

Because thousands of GPUs are attempting to write massive amounts of data over the network simultaneously, a massive I/O bottleneck occurs. During this write phase, the expensive GPUs sit entirely idle. Because this process takes so long, engineers are forced to space out checkpoints—sometimes saving only once every few hours. If a hardware failure happens right before a scheduled save, hours of compute time across thousands of GPUs are permanently lost.

Tier 1: Writing to Local NVMe Drives

Multi-tiered checkpointing eliminates the network bottleneck by introducing a localized storage hierarchy. Instead of sending data straight to the distant centralized storage, the system utilizes the local NVMe (Non-Volatile Memory Express) drives physically installed inside the compute nodes.

NVMe drives offer exceptionally high write speeds and low latency because they communicate directly with the system's CPU and GPUs via the PCIe bus. When it is time to save a checkpoint, the system transfers the model's state from the GPU memory (VRAM) to the system memory (RAM), and then immediately writes it to the local NVMe drive. Because each node writes only its portion of the data to its own dedicated local drive, the process takes seconds rather than minutes.

Asynchronous Processing: Eliminating GPU Idle Time

While writing to local NVMe drives is fast, asynchronous processing makes it virtually invisible to the training timeline. In a synchronous system, the GPUs must stop calculating until the checkpoint is fully written to storage. In an asynchronous system, the saving process happens in the background.

Here is how the asynchronous workflow minimizes lost compute time:

  1. Snapshot to RAM: The GPUs briefly pause to copy the current state into the host system’s CPU memory. This memory-to-memory transfer takes milliseconds.
  2. Resume Training: As soon as the data is in the system RAM, the GPUs immediately go back to training the AI model.
  3. Background Local Write: While the GPUs calculate the next training steps, the CPU manages the process of writing the snapshot from the RAM to the local NVMe drive.
  4. Background Network Flush: Once the data is safely on the local NVMe drive, a background process slowly trickles that data over the network to the centralized storage (Tier 2) without competing with the bandwidth needed for active AI training.

Minimizing Lost Compute During Hardware Failures

The combination of local NVMe speed and asynchronous execution fundamentally changes the math of AI fault tolerance. Because saving a checkpoint no longer interrupts the GPUs for minutes at a time, engineers can increase the frequency of checkpoints dramatically—saving every few minutes instead of every few hours.

When a hardware failure inevitably strikes a node, the cluster does not need to lose hours of progress. If a single GPU crashes, the system can quickly pull the most recent state directly from the fast local NVMe drives of the surviving nodes, and pull the failed node's state from the most recent centralized backup.

By shrinking the time it takes to save a state and removing the penalty of idle GPU time, asynchronous multi-tiered checkpointing ensures that when a failure interrupts an AI run, the system only loses a few minutes of compute time, keeping massive AI projects on schedule and within budget.