How Do Amazon EC2 Capacity Blocks Reserve GPU Clusters?

Amazon EC2 Capacity Blocks for ML allow organizations to reserve highly sought-after GPU instances for specific future time windows to support machine learning workloads. By searching an inventory calendar for available cluster capacity, users can book colocation-optimized compute resources days or weeks in advance. Once purchased, AWS provisions a dedicated Capacity Reservation within an UltraCluster, guaranteeing that the requested GPU instances are available at the scheduled start time and automatically terminating them when the reservation expires.

Understanding EC2 Capacity Blocks for ML

Training foundation models, fine-tuning large language models, and running intensive machine learning experiments require massive, uninterrupted compute power. However, securing on-demand access to high-performance GPU instances like P5, P4d, or Trn1 can be challenging due to high global demand. Simultaneously, committing to multi-year Savings Plans or Reserved Instances is often impractical for short-term bursts, prototyping, or periodic training runs.

Amazon EC2 Capacity Blocks for ML solve this by providing a predictable, time-bound reservation mechanism. Instead of hoping for on-demand availability or locking into long-term contracts, engineering teams can reserve clusters ranging from 1 to 64 instances (handling up to 512 GPUs) for durations between 1 and 182 days.

Co-Location in Amazon EC2 UltraClusters

A defining feature of Capacity Blocks for ML is physical topology optimization. When you reserve a Capacity Block, AWS does not simply allocate isolated instances scattered across an Availability Zone. Instead, all instances within a single Capacity Block are placed together inside an Amazon EC2 UltraCluster.

These instances are interconnected using the Elastic Fabric Adapter (EFA) over a petabit-scale, non-blocking network. This low-latency, high-throughput interconnect is essential for distributed training frameworks like PyTorch Distributed Data Parallel (DDP), DeepSpeed, or Megatron-LM, where collective communication operations such as AllReduce and AllGather frequently become the primary bottleneck across nodes.

Step-by-Step Reservation Mechanics

Reserving and utilizing a GPU cluster through EC2 Capacity Blocks follows a structured lifecycle managed via the AWS Management Console, AWS CLI, or SDKs.

1. Querying Available Capacity Offerings

Unlike standard On-Demand Capacity Reservations that take effect immediately if hardware is free, Capacity Blocks operate on a forward-looking schedule. You submit a search query specifying:

• Instance Type: Such as p5.48xlarge, p4d.24xlarge, p4de.24xlarge, or trn1.32xlarge.
• Instance Count: Typically in increments of 1, 2, 4, 8, 16, 32, or 64 instances.
• Date Range: The earliest start date and latest end date within which you want your training window to occur (up to 8 weeks in advance).
• Duration: The total number of days required.

Using the AWS CLI, you can search for available offerings with the describe-capacity-block-offerings command:

aws ec2 describe-capacity-block-offerings \
    --instance-type p5.48xlarge \
    --instance-count 16 \
    --start-date-range 2026-12-01T00:00:00Z \
    --end-date-range 2026-12-15T00:00:00Z \
    --capacity-duration-hours 168

AWS evaluates its regional scheduling ledger and returns matching offerings. Each offering includes a unique CapacityBlockOfferingId, the specific Availability Zone where the UltraCluster resides, the exact StartDate and EndDate timestamps (standardized to 11:30 UTC), and the upfront fee.

2. Purchasing the Capacity Block

Once a suitable window is identified, you purchase the offering using its ID. Because Capacity Blocks use dynamic pricing based on supply and demand algorithms at the time of booking, the price is locked in upon purchase.

aws ec2 purchase-capacity-block \
    --capacity-block-offering-id cbo-0123456789abcdef0 \
    --instance-platform Linux/UNIX

If we denote the hourly rate per instance as \(R\), the number of instances as \(N\), and the reservation duration in hours as \(H\), the total upfront cost \(C_{\text{total}}\) billed to your account upon purchase is:

\[C_{\text{total}} = N \times H \times R\]

When the purchase succeeds, AWS immediately creates a CapacityReservation resource in the scheduled state. You receive a CapacityReservationId (for example, cr-0abcdef1234567890), which acts as the target identifier for your automated deployment pipelines.

3. State Transitions and Activation

Between the time of purchase and the scheduled window, the reservation remains in the scheduled state. During this phase, no instances can be launched into it.

At the exact scheduled start time (11:30 UTC on the start date), AWS transitions the Capacity Reservation state from scheduled to active and emits an Amazon EventBridge notification:

{
  "version": "0",
  "id": "12345678-1234-1234-1234-123456789012",
  "detail-type": "EC2 Capacity Reservation State Change",
  "source": "aws.ec2",
  "detail": {
    "capacity-reservation-id": "cr-0abcdef1234567890",
    "state": "active"
  }
}

4. Launching Instances into the Reserved Cluster

Once the reservation state is active, you can provision your GPU instances. To ensure your instances consume the reserved UltraCluster capacity, you must explicitly target the CapacityReservationId and set the InstanceMarketOptions market type to capacity-block.

aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --instance-type p5.48xlarge \
    --count 16 \
    --subnet-id subnet-0123456789abcdef0 \
    --InstanceMarketOptions MarketType=capacity-block \
    --capacity-reservation-specification CapacityReservationTarget={CapacityReservationId=cr-0abcdef1234567890}

You can also integrate this CapacityReservationId into AWS Batch, Amazon EKS managed node groups, AWS ParallelCluster, or Amazon SageMaker HyperPod to automate cluster spin-up the moment the EventBridge active event fires.

5. Pre-Expiration Draining and Termination

Because another customer may have booked the same physical UltraCluster hardware for the window immediately following yours, AWS enforces strict boundary times.

Thirty minutes before the reservation's EndDate (at 11:00 UTC), AWS begins the winding-down process:

1. EventBridge Warning: AWS emits an EC2 Capacity Block Expire Warning event via Amazon EventBridge and sends an EC2 Instance rebalance/interruption notice via the Instance Metadata Service (IMDS).
2. Checkpointing Window: Your training scripts have 30 minutes to save model weights and optimizer states to persistent storage such as Amazon S3 or Amazon FSx for Lustre.
3. Automatic Termination: At 11:30 UTC, any instances still running inside the Capacity Block are forcibly terminated by AWS, and the CapacityReservation transitions to the expired state.

Summary of Key Operational Constraints

When architecting ML workflows around EC2 Capacity Blocks, keep the following operational behaviors in mind: