What Happens When an EC2 Dedicated Host Hardware Fails?

When the underlying physical hardware of an Amazon EC2 Dedicated Host experiences an unexpected failure, AWS automatically responds depending on whether Host Recovery is enabled. With Host Recovery enabled, AWS automatically allocates a new replacement Dedicated Host and restarts your affected EC2 instances on the new hardware while retaining their configuration, metadata, and storage volumes. If Host Recovery is disabled, the Dedicated Host enters a permanent failed state, running instances become unreachable or stop, and administrators must manually allocate a new host and relaunch or migrate the instances.

How AWS Detects Dedicated Host Hardware Failures

Amazon EC2 continuously monitors the health of the physical servers backing Dedicated Hosts. Hardware failures can stem from motherboard faults, memory errors, network interface card malfunctions, power supply failures, or underlying hypervisor issues.

When AWS detects an unrecoverable hardware fault on a Dedicated Host:

  1. Health Status Updates: The state of the Dedicated Host transitions to an impaired or failed status in the Amazon EC2 console and API.
  2. AWS Health Dashboard Notifications: AWS publishes an event to the AWS Health Dashboard and emits an event via Amazon EventBridge (formerly CloudWatch Events) alerting you to the host degradation or failure.
  3. Instance State Impact: Instances running on the impaired physical server stop responding to network traffic and fail Instance Status Checks and System Status Checks.

Automatic Recovery with Host Recovery Enabled

Host Recovery is a feature enabled by default on newly allocated Dedicated Hosts for most supported instance families. It minimizes downtime by automating the replacement of failed physical servers.

The Host Recovery Process

When an unexpected hardware failure occurs on a host with Host Recovery enabled, AWS performs the following sequence:

  1. Replacement Host Allocation: AWS automatically provisions a new Dedicated Host of the same instance family and configuration within the same Availability Zone.
  2. Instance Restart: Affected instances are automatically rebooted onto the new replacement Dedicated Host.
  3. Configuration Preservation: The recovered instances retain their original:
    • Instance ID
    • Private IPv4 and IPv6 addresses
    • Elastic IP addresses
    • Amazon EBS volume attachments
    • Instance metadata and tags
  4. Old Host Retirement: The failed Dedicated Host is automatically released from your account once recovery completes, ensuring you are not billed for the impaired hardware.

Note that the new replacement Dedicated Host receives a new Dedicated Host ID, while the instances themselves keep their existing Instance IDs.

Limitations of Host Recovery

While Host Recovery automates continuity, there are specific constraints to consider:

  • Instance Store Volumes: Any data stored on ephemeral NVMe or HDD instance store volumes is lost during hardware failure because instance store media is physically attached to the failed host. Only data on Amazon EBS volumes persists.
  • Host Capacity Availability: Recovery depends on the availability of spare physical capacity for that specific instance family in the target Availability Zone. If AWS lacks sufficient capacity at the exact moment of failure, automatic recovery may be delayed until capacity becomes available.
  • Unsupported Configurations: Host Recovery is not supported for Dedicated Hosts configured to support multiple instance types within the same instance family unless supported by the specific hardware generation, nor is it supported on certain bare metal (metal) or macOS Dedicated Hosts.

What Happens When Host Recovery Is Disabled

If Host Recovery is turned off, or if the Dedicated Host type does not support automatic recovery, AWS does not allocate a replacement host automatically.

  1. Instance Interruption: Running EBS-backed instances on the failed host transition to a stopped state or remain stuck in a running state while failing status checks. Instance store-backed instances are terminated.
  2. Manual Replacement: You must manually allocate a new Dedicated Host in your account using the EC2 console, AWS CLI, or SDK.
  3. Changing Affinity and Starting Instances:
    • If the instances had their affinity set to a specific Host ID, you must modify the instance placement attributes to point to the newly allocated Host ID before starting them.
    • If the instances belong to a Host Resource Group and have affinity set at the group level, starting the stopped instances will automatically place them onto any available host within that resource group.
  4. Releasing the Failed Host: You must manually release the impaired Dedicated Host once your instances have been migrated off it (or AWS will eventually retire and decommission the failed host after sending retirement notifications).

Impact on Software Licensing

Organizations typically use EC2 Dedicated Hosts to satisfy Bring Your Own License (BYOL) requirements bound to physical cores or sockets (such as Windows Server, SQL Server, Oracle, or RHEL).

When a host fails and is replaced:

  • New Host ID: Because the replacement server is a different physical machine, it has a new Host ID.
  • AWS License Manager Integration: If you manage your Dedicated Hosts using AWS License Manager and Host Resource Groups, License Manager automatically tracks the release of the failed host and associates licenses with the newly allocated replacement host.
  • License Mobility and Re-binding: If your software licensing is strictly bound to hardware identifiers (like UUIDs or physical MAC addresses) rather than core/socket counts tracked via AWS License Manager, you may need to reactivate or re-bind the license on the replacement host.

Availability and Recovery Time Calculations

When designing high-availability architectures on Dedicated Hosts, engineers account for the recovery time of a single host vs. multi-AZ deployments. If a single Dedicated Host has an annual failure rate producing an availability \(A_{host}\), the probability of simultaneous failure across \(n\) independent Dedicated Hosts in separate Availability Zones is given by:

\[P(\text{all fail}) = (1 - A_{host})^n\]

Consequently, overall cluster availability \(A_{cluster}\) across \(n\) redundant hosts is:

\[A_{cluster} = 1 - (1 - A_{host})^n\]

Because Host Recovery requires cold-booting instances onto a new physical server, the Recovery Time Objective (RTO) for a single host failure typically ranges from several minutes to 15 minutes. Workloads requiring near-zero RTO should distribute instances across multiple Dedicated Hosts in different Availability Zones rather than relying solely on single-host recovery.

Best Practices for Dedicated Host Resiliency

To minimize disruption from unexpected physical hardware failures on EC2 Dedicated Hosts:

  • Enable Host Recovery: Ensure Host Recovery is enabled on all eligible Dedicated Hosts via the EC2 console or by specifying --host-recovery on in the allocate-hosts or modify-hosts CLI commands.
  • Use Amazon EBS for Persistent Data: Store all critical application state and databases on Amazon EBS volumes or networked storage (such as Amazon EFS or Amazon FSx) rather than local instance store volumes.
  • Deploy Across Multiple Availability Zones: Spread Dedicated Hosts across at least two Availability Zones so that a physical server failure does not cause a complete service outage while Host Recovery restarts the affected instances.
  • Use Host Resource Groups: Combine AWS License Manager with Host Resource Groups and Auto Placement so instances can seamlessly restart on healthy hosts without requiring manual Host ID reassignments.
  • Monitor AWS Health Events: Configure Amazon EventBridge rules to capture AWS_EC2_DEDICATED_HOST_EVENT notifications and trigger automated alerts or failover workflows when hardware degradation occurs.