Which EC2 instance family is best for MapReduce and HDFS?

Amazon EC2 D series instances, such as the D2 and D3 families, are the Storage Optimized instances specifically tailored for MapReduce and distributed file systems like the Hadoop Distributed File System. These instances provide dense local hard disk drive storage, delivering massive sequential read and write throughput across large datasets. By pairing high-capacity local storage with high network bandwidth, D series instances lower the cost per terabyte of disk storage while meeting the input and output demands of big data analytics and massively parallel processing workloads.

Overview of EC2 Storage Optimized Instance Families

Amazon Elastic Compute Cloud (EC2) categorizes instances with high local disk throughput and IOPS under the Storage Optimized umbrella. While families like the I series (such as I3 and I4i) focus on low-latency, high random I/O performance using Non-Volatile Memory Express (NVMe) solid-state drives (SSDs) for NoSQL databases, workloads like MapReduce and HDFS have a fundamentally different access pattern. Distributed file systems and batch data processing pipelines read and write large contiguous blocks of data sequentially rather than performing small, random lookups.

To support high-throughput sequential access cost-effectively, AWS designed the D series instance families:

D2 Instances: Built for dense HDD storage, offering up to 48 TB of local HDD-based instance store capacity across 24 individual 2 TB drives.
D3 and D3en Instances: The modern successor to D2, built on the AWS Nitro System. D3 instances provide higher compute performance, up to 75 Gbps of network bandwidth, and up to 336 TB of local HDD storage on D3en configurations.

Why MapReduce and HDFS Require Dense HDD Storage

MapReduce and HDFS operate on the principle of data locality: moving computation to the nodes where the data resides rather than moving massive datasets across the network to separate compute nodes. In a typical HDFS cluster, files are split into large blocks, often 128 MB or 256 MB in size, and replicated across multiple DataNodes.

Because these blocks are large and stored contiguously, magnetic hard disk drives (HDDs) perform exceptionally well. Once the disk read/write head seeks to the beginning of an HDFS block, it can stream data continuously at high sequential speeds. Using multi-terabyte HDD arrays directly attached to the host server satisfies the two primary hardware requirements of distributed file systems:

High Sequential Disk Throughput: When a MapReduce job runs, the Map phase scans vast amounts of input data stored locally, while the Shuffle and Reduce phases write intermediate and final outputs back to disk.
Low Cost per Terabyte: Big data lakes and historical log repositories frequently scale into petabytes. Because HDFS handles fault tolerance in software by replicating blocks (typically with a replication factor of \(R = 3\)), the raw storage required is three times the usable data volume. If a dataset has a logical size of \(S_{\text{logical}}\), the physical cluster storage capacity \(S_{\text{physical}}\) required is:

\[S_{\text{physical}} = S_{\text{logical}} \times R\]

For example, storing \(100\text{ TB}\) of usable data with standard replication requires \(300\text{ TB}\) of raw disk space. Local HDD storage on D2 or D3 instances provides this capacity at a fraction of the cost of SSD-backed instances.

Key Features of the D2 and D3 Instance Families

High Sequential Read and Write Throughput

D series instances aggregate the performance of multiple physical hard drives attached directly to the physical host. On the largest D3en instance (d3en.12xlarge), 24 individual 14 TB HDDs work in parallel to deliver up to 4.6 GB/s of aggregate sequential disk throughput at a 2 MiB block size. If each drive delivers a sequential read speed of \(T_{\text{drive}}\), the aggregate throughput \(T_{\text{total}}\) across \(N\) independent drives scales linearly:

\[T_{\text{total}} = \sum_{i=1}^{N} T_{\text{drive}, i} \approx N \times T_{\text{drive}}\]

In distributed file systems like HDFS or GPFS, each disk is mounted as a separate volume (a configuration known as JBOD, or "Just a Bunch of Disks"), allowing the distributed file system software to manage block placement, parallel I/O, and drive failure recovery directly without RAID controller overhead.

High Network Bandwidth for the Shuffle Phase

While the Map phase relies heavily on local disk reads, the Shuffle phase of a MapReduce job transfers intermediate key-value pairs across all nodes in the cluster. Network bottlenecks during the Shuffle phase can stall an entire job. D3 instances offer up to 25 Gbps of network bandwidth, while D3en instances provide up to 75 Gbps, ensuring that inter-node data transfers keep pace with local disk throughput.

Instance Store Persistence Considerations

The high-density HDDs provided by D2 and D3 instances are ephemeral instance store volumes. Data written to these drives persists during instance reboots, but it is permanently lost if the underlying hardware fails or if the instance is stopped or terminated. In an HDFS architecture, this behavior is expected and managed at the application layer: if an EC2 instance fails, the HDFS NameNode detects the missing DataNode and automatically re-replicates its blocks from healthy nodes elsewhere in the cluster.

Common Use Cases Beyond Hadoop

In addition to Apache Hadoop and HDFS, the D2 and D3 Storage Optimized instance families are tailored for a variety of distributed and data-intensive workloads:

Massively Parallel Processing (MPP) Data Warehouses: Analytical databases that partition data across local disks and execute parallel table scans.
Distributed File Systems: Red Hat GlusterFS, Lustre, and IBM Spectrum Scale (GPFS) deployments that pool local storage across nodes into a unified namespace.
Log and Data Processing Pipelines: High-volume ingestion and batch processing systems using frameworks like Apache Spark, where large datasets are cached and spilled to local disks during complex transformations.