What Happens When an EC2 Capacity Block for ML Expires?

When an Amazon EC2 Capacity Block for Machine Learning reaches the end of its reservation period, AWS automatically terminates all running instances within the reservation and changes the Capacity Block state to expired. Thirty minutes before the reservation ends, Amazon EC2 begins the termination process, sending an interruption notice via Amazon EventBridge to allow your workloads to save checkpoints and gracefully shut down before the capacity is reclaimed at the exact end time.

The Expiration Timeline

Amazon EC2 Capacity Blocks for ML are designed to provide predictable, fixed-duration access to GPU instances. Because this capacity is often scheduled for other customers immediately following your reservation, the end time is strictly enforced. The expiration process follows two distinct phases:

  1. 30 Minutes Before Expiration (\(t_{end} - 30\text{ minutes}\)):
    • The status of the Capacity Block transitions to expiring.
    • You can no longer launch new instances into the Capacity Block during this final 30-minute window.
    • Amazon EC2 emits an instance interruption warning via Amazon EventBridge and makes the notice available through the Instance Metadata Service (IMDS).
    • Amazon EC2 begins draining and terminating instances running inside the Capacity Block.
  2. At the Expiration Time (\(t_{end}\)):
    • The status of the Capacity Block transitions to expired.
    • Any remaining instances that have not yet finished shutting down are forcibly terminated by AWS.
    • The underlying compute capacity is released back to AWS.

If we denote the scheduled end time as \(t_{end}\), the guaranteed uninterrupted operational window \(W\) for your instances is governed by:

\[W = [t_{start}, t_{end} - 30\text{ minutes}]\]

What Happens to Your Instances and Storage?

Unlike standard On-Demand Capacity Reservations—where instances continue running as regular On-Demand instances after the reservation expires—instances inside a Capacity Block for ML cannot continue running past the reservation window.

  • EC2 Instances: All instances targeting the Capacity Block are permanently terminated, regardless of their configured shutdown behavior. You cannot stop and restart them once the block expires.
  • Instance Store Volumes: Any data stored on local NVMe instance store volumes is permanently erased when the instances terminate.
  • Amazon EBS Volumes: By default, root EBS volumes with the DeleteOnTermination attribute set to true are deleted. Attached data volumes with DeleteOnTermination set to false will detach and remain in your account, preserving your data.
  • Shared Storage (Amazon S3, FSx for Lustre, EFS): Data written to external storage services remains completely intact and is the recommended destination for model checkpoints and training artifacts.

Monitoring Expiration Events

To prevent data loss during distributed training or inference workloads, you should configure automated workflows to listen for expiration events. Amazon EC2 publishes events to Amazon EventBridge at key intervals:

  • Capacity Block Expiration Warning: Sent when the Capacity Block enters the expiring state 30 minutes prior to \(t_{end}\).
  • EC2 Instance Rebalance / Interruption Notice: Sent to individual instances as they are scheduled for termination.

You can capture these events using an EventBridge rule or by polling the instance metadata endpoint locally from your training nodes:

curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/events/recommendations/rebalance

Best Practices Before Your Capacity Block Ends

To ensure a smooth transition when your Capacity Block reaches the end of its reservation period, implement the following practices:

  1. Schedule Final Checkpoints Early: Configure your training scripts to write their final model checkpoints to Amazon S3 or Amazon FSx for Lustre prior to \(t_{end} - 30\text{ minutes}\). Do not wait until the final minute of the reservation, as instance termination begins 30 minutes before the official end time.
  2. Automate Graceful Shutdowns: Use Amazon EventBridge rules triggered by the expiring state change to invoke AWS Systems Manager Run Command or a lifecycle hook in your orchestrator (such as Amazon EKS or Slurm) to flush logs and save intermediate states.
  3. Preserve Persistent EBS Volumes: Verify that any EBS volumes holding critical datasets or checkpoints have DeleteOnTermination disabled before the termination window begins:
aws ec2 modify-instance-attribute \
    --instance-id i-1234567890abcdef0 \
    --block-device-mappings "[{\"DeviceName\": \"/dev/sdf\",\"Ebs\":{\"DeleteOnTermination\":false}}]"
  1. Extend or Book Follow-up Capacity in Advance: If you anticipate needing more time, you can check if an extension is available for your existing Capacity Block using the AWS Management Console or AWS CLI (describe-capacity-block-extension-offerings). If an extension is available and purchased before the block expires, your instances will continue running seamlessly into the new window without interruption.