Why Use Amazon FSx for Lustre with EC2 ML Clusters?

A high-performance machine learning cluster attaches Amazon FSx for Lustre to its Amazon EC2 compute nodes to eliminate data input and output bottlenecks during distributed training. By providing sub-millisecond latencies, hundreds of gigabytes per second of throughput, and millions of IOPS, FSx for Lustre ensures that expensive GPU and accelerator instances remain fully saturated with training data rather than idling while waiting for storage reads. Furthermore, its native integration with Amazon S3 allows clusters to seamlessly hydrate datasets on demand and persist model checkpoints back to durable object storage through a POSIX-compliant shared file system.

Eliminating GPU Starvation in Distributed Training

Modern deep learning workloads rely on clusters of high-performance Amazon EC2 instances, such as P4d, P5, or Trn1 nodes, working in parallel across terabytes or petabytes of training data. While the compute capacity of GPUs and custom ML accelerators has scaled rapidly, storage throughput often lags behind, creating a condition known as GPU starvation. When compute nodes process mini-batches faster than the underlying storage system can deliver them, expensive accelerators sit idle waiting on I/O operations.

In a distributed data-parallel training job across \(N\) compute nodes, each node \(i\) with batch processing rate \(R_i\) (measured in samples per second) and average sample size \(S\) (in bytes) generates an aggregate read bandwidth demand \(B_{\text{total}}\) defined as:

\[B_{\text{total}} = \sum_{i=1}^{N} R_i \cdot S\]

If the storage layer provides a maximum throughput \(B_{\text{storage}} < B_{\text{total}}\), the overall training step time is bounded by I/O latency rather than compute efficiency. Amazon FSx for Lustre resolves this bottleneck by striping file data across multiple Object Storage Targets (OSTs), enabling parallel access across the entire EC2 cluster. Throughput scales linearly with storage capacity, reaching hundreds of gigabytes per second and millions of IOPS with consistent sub-millisecond latencies.

Native Amazon S3 Data Repository Integration

Machine learning pipelines typically store raw training datasets, feature sets, and historical checkpoints in Amazon S3 due to its durability, low cost, and integration with data lakes. However, accessing millions of small files (such as individual images, audio clips, or tokenized shards) directly from S3 over HTTP REST APIs during every training epoch introduces significant request latency and API overhead.

Amazon FSx for Lustre bridges the gap between object storage and high-performance compute by linking directly to Amazon S3 buckets as Data Repository Associations (DRAs). When linked to an S3 bucket, FSx for Lustre operates as a high-speed, transparent caching layer:

  1. Lazy Loading of Metadata and Data: Upon creation or synchronization, FSx for Lustre imports object metadata from S3 into its Metadata Targets (MDTs) without immediately copying the entire dataset. Compute nodes see the complete directory structure immediately.
  2. First-Read Hydration: When an EC2 compute node requests a file for the first time during epoch one, FSx for Lustre automatically fetches the object from S3 in parallel and caches it on fast NVMe SSD storage.
  3. Sub-Millisecond Subsequent Epochs: During subsequent training epochs, all read requests are served directly from the low-latency Lustre file system, bypassing S3 network overhead entirely.
  4. Data Export: As training progresses, model weights, logs, and checkpoints written to the Lustre file system by EC2 nodes can be asynchronously or manually exported back to the linked S3 bucket for long-term retention.

Shared POSIX Compliance Across Compute Nodes

Distributed machine learning frameworks like PyTorch, TensorFlow, DeepSpeed, and Horovod expect standard POSIX file system semantics for reading datasets and coordinating checkpoints across worker nodes. Rewriting training scripts to use custom object-storage SDKs or managing local NVMe instance store synchronization across dozens of nodes adds operational complexity and brittle boilerplate code.

Because Amazon FSx for Lustre is fully POSIX-compliant, multiple EC2 instances can mount the exact same file system concurrently under a unified namespace (such as /fsx). This shared architecture provides several operational advantages:

  • Zero Code Modification: Existing training scripts that use standard file I/O libraries (open(), os.listdir(), torch.load()) work immediately without modification.
  • Elimination of Data Duplication: Instead of copying a 10 TB dataset onto the local EBS volumes or NVMe drives of 64 separate EC2 instances—which wastes 640 TB of storage and delays cluster startup—every node reads from a single shared dataset.
  • Unified Checkpointing: During large language model (LLM) training, worker nodes must periodically save distributed state shards. A shared POSIX mount allows all ranks to write their respective shards to a single shared directory simultaneously and allows any replacement node to immediately resume from the latest checkpoint if an EC2 instance fails.

High-Throughput Networking with EFA and GPUDirect Storage

To maximize the performance between Amazon EC2 compute nodes and the storage cluster, Amazon FSx for Lustre supports Elastic Fabric Adapter (EFA) and NVIDIA GPUDirect Storage (GDS) on supported instance types.

In traditional storage architectures, reading data from a network file system requires copying buffers through the host CPU and system memory (DRAM) before transferring them over the PCIe bus into GPU memory (VRAM). By combining EFA with GPUDirect Storage, FSx for Lustre enables a direct Direct Memory Access (DMA) data path between the network interface card and the GPU memory. This bypasses the host CPU bounce buffer, lowers CPU utilization on the EC2 nodes, reduces I/O latency, and maximizes effective PCIe bandwidth for gradient synchronization and tensor computation.

Flexible Deployment Options for ML Lifecycles

Machine learning workloads vary between short-lived burst training runs and long-running research clusters. Amazon FSx for Lustre provides two primary deployment types to optimize cost and performance for EC2 clusters:

  • Scratch File Systems: Designed for temporary storage and short-term processing of data. Scratch file systems do not replicate data across servers, allowing them to deliver the highest burst throughput per terabyte at the lowest cost. Because the authoritative dataset lives in Amazon S3, a scratch file system can be spun up when an EC2 training cluster launches, hydrated from S3, and torn down immediately after training completes and checkpoints are exported.
  • Persistent File Systems: Designed for long-term storage and continuous multi-day or multi-week training workloads. In persistent deployments, data is replicated within the Availability Zone, and failed storage servers are replaced automatically and transparently within minutes, ensuring that long-running foundational model training jobs do not fail due to storage hardware interruptions.