What Happens When an EC2 Dense Storage Disk Fails?

When a physical disk fails on a multi-drive dense storage instance in Amazon EC2, the affected instance store volume becomes inaccessible and any data stored exclusively on that drive is lost. Unlike Amazon Elastic Block Store volumes, which are network-attached and automatically replicated across multiple servers within an Availability Zone, EC2 instance store volumes reside directly on the physical disks attached to the host computer. Despite the loss of a single drive, the EC2 instance itself continues to run normally, and all other healthy drives attached to the instance remain operational unless the failed disk was part of an unredundant array or held critical operating system files. Because hardware failures are an expected reality in large-scale infrastructure, architectures relying on dense storage instances such as the d2, d3, or i3 families must implement software-level redundancy, distributed file systems, or regular automated backups to prevent permanent data loss and service interruption.

Understanding EC2 Dense Storage Instances

Amazon EC2 dense storage instances, such as the D2, D3, and D3en instance families, are designed for workloads that require high sequential read and write access to very large datasets. These instances come equipped with multiple locally attached hard disk drives (HDDs) or non-volatile memory express (NVMe) solid-state drives (SSDs) known as instance store volumes.

Instance store volumes provide temporary block-level storage. Because the drives are physically attached to the underlying host server, they avoid the network latency associated with Amazon Elastic Block Store (EBS) and deliver high random and sequential I/O performance at a lower cost per gigabyte. However, this direct hardware attachment means there is no built-in AWS abstraction layer replicating your data behind the scenes. Each mounted instance store device maps directly to a physical drive on the host rack.

Immediate Impact of a Single Drive Failure

When one of the physical drives in a multi-drive instance experiences a hardware failure, several immediate events occur across the operating system and the AWS management plane:

  1. Input/Output Errors: The operating system kernel begins logging I/O errors (such as blk_update_request: I/O error in Linux dmesg or /var/log/messages) whenever an application attempts to read from or write to the affected block device.
  2. File System Degradation: If the drive is formatted with an independent file system (such as XFS or ext4), the kernel will typically remount the file system as read-only to prevent further corruption, or the mount point will become completely unresponsive.
  3. Instance Continuity: A single failed instance store volume does not cause the EC2 instance to stop, reboot, or terminate, provided the root volume is hosted on Amazon EBS (which is the standard configuration for modern dense storage instances). The remaining healthy physical drives continue to function without performance degradation.
  4. AWS Health Notifications: AWS detects the underlying hardware fault on the host and posts an event to the AWS Health Dashboard and Amazon EventBridge. While AWS notifies you of the degraded hardware, AWS technicians cannot hot-swap a physical drive in a running instance without affecting your workload, meaning remediation requires action from the user.

How Storage Configuration Determines Data Survival

The exact impact of a disk failure on your applications depends entirely on how the multiple drives are configured within the operating system.

Independent Mount Points (JBOD)

In a "Just a Bunch of Disks" (JBOD) configuration, each block device (for example, /dev/nvme1n1, /dev/nvme2n1, etc.) is formatted and mounted to its own directory, such as /data1, /data2, and /data3.

When a drive fails in a JBOD setup, only the data on that specific mount point is lost. Distributed data stores such as Apache Hadoop (HDFS), Apache Kafka, and Elasticsearch are specifically designed to use JBOD configurations. They handle disk failures natively by replicating data chunks or shards across multiple disks and multiple EC2 instances. If one drive fails, the application detects the dead mount point, stops routing requests to it, and serves reads from replica nodes.

RAID 0 (Striping)

Many workloads combine multiple instance store volumes into a single logical volume using Linux mdadm or Logical Volume Manager (LVM) in a RAID 0 array to maximize throughput and aggregate storage capacity. In RAID 0, data blocks are striped across all \(N\) drives in the array with no parity or mirroring.

If a single disk fails in a RAID 0 array, the entire logical volume fails and 100% of the data across all drives in the array is lost. Furthermore, the probability of experiencing a failure increases with the number of drives. If a single drive has a probability \(p\) of failing over a given period, the probability \(P_{\text{fail}}\) of a RAID 0 array with \(N\) drives failing is:

\[P_{\text{fail}} = 1 - (1 - p)^N\]

On a d3.8xlarge instance with 24 physical drives, a RAID 0 array is significantly more vulnerable to failure than a single disk. RAID 0 should only be used when the instance holds purely ephemeral scratch data or when the instance is part of a larger cluster that replicates data across multiple EC2 nodes.

RAID 1, RAID 5, RAID 6, and RAID 10

To protect against single-drive failures at the operating system level without relying on multi-node application replication, administrators configure redundant software RAID arrays:

  • RAID 1 / RAID 10: Mirrors data across drive pairs. If a disk fails, the array operates in a degraded state with virtually no performance penalty, reading from the surviving mirror drive.
  • RAID 5: Stripes data and parity across at least three drives. It can survive the loss of \(1\) drive. For an array of \(N\) drives each of capacity \(C\), the usable capacity \(U\) is:

\[U = (N - 1) \times C\]

  • RAID 6: Stripes data and dual parity across at least four drives. It can survive the simultaneous loss of \(2\) drives, yielding a usable capacity of:

\[U = (N - 2) \times C\]

When a disk fails in a RAID 1, 5, 6, or 10 array, the logical file system remains online and accessible, though RAID 5 and RAID 6 arrays will experience degraded read and write performance because missing blocks must be reconstructed on the fly using parity calculations.

How to Recover from a Failed Instance Store Disk

Because you cannot replace an individual physical hard drive on an active EC2 host without moving the instance to new hardware, recovery requires migrating the workload to a healthy underlying host server.

1. Isolate the Failed Drive (If Continuing Temporarily)

If your instance is part of a distributed cluster (like HDFS or Cassandra) and can tolerate running with \(N - 1\) drives until a scheduled maintenance window, unmount the failed drive and remove it from your application's configuration so the OS stops attempting I/O operations against the dead device.

2. Back Up Surviving Data

Before changing the instance state, ensure that any critical data remaining on healthy drives (or on a degraded RAID array) is backed up to Amazon S3, Amazon EFS, or an Amazon EBS volume.

3. Stop and Start the Instance (Or Replace via Auto Scaling)

To get a fresh set of healthy physical disks, the instance must be moved to a new physical host:

  • Stop and Start: Perform a Stop action followed by a Start action from the EC2 console or CLI. Stopping and starting an EBS-backed instance migrates the virtual machine to a new physical host computer with a completely new set of healthy instance store drives. Note: Simply issuing an OS-level Reboot will not fix the problem, because a reboot keeps the instance on the exact same physical host with the broken drive. Furthermore, stopping an instance erases all data on all attached instance store volumes.
  • Instance Replacement: In production environments, the preferred approach is to terminate the degraded instance and launch a replacement instance (either manually or via an EC2 Auto Scaling group), allowing your cluster software or automated bootstrap scripts to format the new drives and re-sync the required data from peer nodes or Amazon S3.