Why Use EC2 Instance Store for HPC Scratch Data and Caches?

High-performance computing workloads use Amazon EC2 instance store volumes for temporary scratch data and caches to achieve ultra-low latency and maximum input/output operations per second. Because these volumes are physically attached to the host server rather than connected over a network, they eliminate network bottlenecks and provide rapid data access for compute-intensive tasks. In addition, instance store storage is included in the hourly cost of the EC2 instance, offering a cost-effective solution for managing massive volumes of transient data that do not require long-term persistence.

Understanding Amazon EC2 Instance Store

Amazon EC2 instance store provides temporary block-level storage for an EC2 instance. Unlike Amazon Elastic Block Store (EBS), which operates over the network as remote storage, instance store volumes reside on disks that are physically attached to the host computer running the virtual machine. Most modern instance families equipped with instance store utilize Non-Volatile Memory Express (NVMe) Solid State Drives (SSDs) connected directly to the host PCIe bus.

Because the storage is tied directly to the underlying physical hardware, the lifecycle of the data on an instance store volume matches the lifecycle of the EC2 instance. Data remains intact if the operating system reboots, as the instance stays on the same physical host. However, if the instance is stopped, hibernated, or terminated, or if the underlying physical drive fails, every block of data on the instance store volume is permanently lost. This ephemeral nature makes it unsuitable for primary databases or critical long-term state, yet ideal for disposable data generated during active computation.

Ultra-Low Latency and High IOPS

In High-Performance Computing (HPC), processors frequently wait on storage subsystems to feed them data or write out intermediate states. Network-attached block storage must traverse the virtualization layer and the storage network stack, introducing microsecond-level delays that compound over millions of operations.

Physically attached NVMe instance store volumes bypass the network layer entirely. When an application requests a read or write operation, the request travels directly over the local PCIe lanes to the storage media. This architecture delivers sub-millisecond latency—often measured in tens of microseconds—and enables a single instance to achieve millions of Input/Output Operations Per Second (IOPS).

The relationship between latency, concurrency, and throughput can be expressed using Little's Law applied to storage queues:

\[\text{IOPS} = \frac{\text{Queue Depth}}{\text{Latency}}\]

When latency (\(L\)) decreases significantly, an HPC workload can achieve much higher \(\text{IOPS}\) at a lower queue depth, preventing CPU threads from blocking while waiting for disk responses. For multi-disk instances, administrators frequently combine multiple NVMe instance store volumes into a single logical volume using a RAID 0 (striping) configuration. If an instance has \(N\) physical NVMe drives, each capable of a throughput \(T_{\text{drive}}\), the aggregate sequential throughput \(T_{\text{total}}\) approaches:

\[T_{\text{total}} \approx N \times T_{\text{drive}}\]

This allows large storage-optimized or compute-optimized instances to reach tens of gigabytes per second in sequential read and write bandwidth.

Preserving Network Bandwidth for Inter-Node Communication

Distributed HPC workloads—such as computational fluid dynamics, weather modeling, and molecular dynamics—rely heavily on Message Passing Interface (MPI) to communicate across hundreds or thousands of compute nodes. These nodes exchange boundary conditions and synchronization messages continuously over high-speed network interfaces like the AWS Elastic Fabric Adapter (EFA).

On many cloud architectures, network-attached storage traffic and inter-node network traffic share the same physical network interface card or underlying host bandwidth allocation. If a workload writes terabytes of temporary scratch files to network-attached storage, that storage traffic can contend with MPI communication or traffic heading to shared parallel file systems like Amazon FSx for Lustre. By routing local scratch reads and writes to the physical PCIe bus via instance store volumes, the workload keeps local I/O off the network interface. This isolation ensures that 100% of the available network bandwidth remains dedicated to node-to-node communication and shared cluster coordination.

Cost Efficiency at Scale

Cost management is a major factor when scaling HPC clusters to thousands of cores. With network-attached storage like Amazon EBS, organizations pay separately for the provisioned storage capacity (per gigabyte-month) and, depending on the volume type, for provisioned IOPS and throughput. Provisioning multiple terabytes of high-IOPS network storage across hundreds of nodes for a short-lived simulation can quickly eclipse the cost of the compute instances themselves.

Instance store volumes are included in the base hourly price of the EC2 instance. Whether an HPC job writes a few megabytes or saturates the local NVMe drives with terabytes of temporary calculations, there are no per-GB or per-IOPS charges for the local storage. For workloads with high data churn—where massive datasets are written, read once, and discarded—using instance store eliminates storage provisioning overhead and significantly lowers the total cost per simulation run.

Common HPC Use Cases for Instance Store

Because of its performance profile and ephemeral nature, HPC architects target specific data patterns for EC2 instance store volumes:

  • Out-of-Core Computation Scratch Space: Solvers in finite element analysis (FEA) and seismic processing often generate matrices too large to fit into system RAM. When memory capacity (\(M_{\text{RAM}}\)) is exceeded by the problem size (\(S_{\text{matrix}}\)), the solver spills intermediate matrix blocks to disk. High-speed NVMe scratch space minimizes the performance penalty of this out-of-core execution.
  • Local Dataset Caching for Machine Learning: Deep learning training jobs frequently iterate over millions of small files (such as images or audio clips) stored in Amazon S3. Fetching these files repeatedly over the network introduces bottlenecks during each training epoch. Workloads copy and unpack the training dataset onto the local instance store at the start of the job, allowing GPUs to read data at local NVMe speeds for all subsequent epochs.
  • Genomics Alignment and Sorting: Secondary genomics analysis involves splitting raw sequencing reads, aligning them against a reference genome, and sorting massive intermediate BAM or SAM files. Storing these temporary split chunks on local instance store speeds up the sort and merge phases before the final variant calls are written back to durable storage.
  • Burst Buffer and Staging Tier: Before committing checkpoints to a shared parallel file system or object store, compute nodes can write application state dumps rapidly to their local instance store. The simulation immediately resumes computation on the CPUs while a background process asynchronously trickles the checkpoint data from the instance store to durable storage like Amazon S3 or Amazon FSx.

Architectural Best Practices for Ephemeral Storage

To use EC2 instance store effectively without risking valuable research or production results, workloads must be designed around data disposability:

  1. Automated Initialization: Because instance store volumes arrive unformatted or wiped upon fresh instance launch, launch scripts or cluster managers (such as AWS ParallelCluster) should automatically format the NVMe devices with a high-performance file system like XFS or ext4 and mount them to a designated /scratch or /tmp directory at boot time.
  2. Tiered Checkpointing: Never store the only copy of a long-running simulation's checkpoint on an instance store volume. If a Spot Instance is interrupted or a hardware failure occurs, all progress is lost. Use the instance store as a staging area and replicate critical checkpoints to persistent storage at regular intervals.
  3. Final Output Ingestion: Configure job orchestration pipelines so that the final step of any batch job copies the synthesized results from the local instance store to a durable destination before the instance is terminated.