When to Handle the EC2 Spot Instance Rebalance Signal?

An application should handle the Amazon EC2 Spot Instance Rebalance Recommendation signal when its graceful shutdown, state checkpointing, or node replacement processes require more than the standard two minutes provided by the Spot Instance interruption notice. While the standard termination notice gives a strict two-minute countdown before an instance is reclaimed, the rebalance recommendation arrives proactively when AWS detects an elevated risk of interruption. This early warning allows distributed systems, container orchestrators, and stateful workloads to provision replacement capacity and drain workloads safely without rushing against a hard two-minute deadline.

Understanding the Rebalance Recommendation Signal

Amazon EC2 Spot Instances offer unused EC2 capacity at a significant discount compared to On-Demand pricing, with the trade-off that AWS can reclaim the capacity when demand rises. Historically, applications relied solely on the Spot Instance interruption notice, which is emitted exactly two minutes before termination.

To improve workload continuity, AWS introduced the EC2 Instance Rebalance Recommendation signal. Amazon EC2 emits this signal when a Spot Instance is at an elevated risk of interruption, even if an actual reclamation has not yet been scheduled. In many cases, the rebalance signal arrives several minutes or more before a two-minute interruption notice, though it is also possible for an instance to receive a rebalance recommendation and never actually be interrupted, or in rare cases of rapid capacity shifts, receive the interruption notice simultaneously.

Key Scenarios for Handling the Rebalance Signal

Not every workload needs to act on the rebalance signal. If a stateless microservice can drain connections and shut down in 15 seconds, waiting for the standard two-minute interruption notice is often sufficient and avoids unnecessary instance churn. However, handling the rebalance recommendation is critical in several specific scenarios.

Long Graceful Shutdown and Connection Draining

Applications that maintain long-lived connections or process lengthy requests often need more than two minutes to wrap up cleanly. Examples include:

  • WebSocket servers or streaming media relays that must migrate active client sessions without dropping frames or connections.
  • API gateways or load-balanced backend services handling complex, synchronous transactions that take over a minute to complete.
  • Queue workers processing single jobs that take several minutes per message and cannot be easily paused mid-execution.

By acting on the rebalance recommendation, the instance can immediately deregister from the load balancer or stop pulling new messages from the queue while finishing in-flight tasks well ahead of any forced termination.

Slow Replacement Provisioning and Cold Starts

Even if an application shuts down quickly, spinning up a replacement node to maintain cluster capacity might take longer than two minutes. You should handle the rebalance signal when:

  • New EC2 instances require several minutes to boot, run user data scripts, download large dependencies, or warm up local caches.
  • Containerized workloads on Amazon EKS or Amazon ECS need to pull large container images (such as machine learning models) onto a new node before pods can transition to a ready state.
  • Your architecture requires zero capacity degradation, meaning the replacement instance must be healthy and serving traffic before the at-risk instance is terminated.

If the time required to provision a replacement instance (\(T_{\text{provision}}\)) plus the time required to drain the existing instance (\(T_{\text{drain}}\)) exceeds the standard two-minute interruption window (\(120\text{ seconds}\)), relying solely on the interruption notice creates a capacity gap:

\[T_{\text{provision}} + T_{\text{drain}} > 120\text{ seconds}\]

Handling the rebalance signal allows you to start \(T_{\text{provision}}\) proactively so the new instance is ready before the old one disappears.

Stateful Workloads and Large Checkpoints

Workloads that store intermediate state in memory or on local ephemeral NVMe storage (Instance Store) must persist that data to durable storage like Amazon S3 or Amazon EBS before termination. You should use the rebalance recommendation when:

  • Big data frameworks (such as Apache Spark or Presto) need extra time to decommission nodes and shuffle blocks of data to other executors in the cluster.
  • Machine learning training jobs must write multi-gigabyte model checkpoints to Amazon S3, where network throughput limits make a two-minute upload risky.
  • High-performance computing (HPC) simulations need to synchronize state across multiple nodes at a specific barrier step before saving progress.

How to Detect the Rebalance Signal

AWS exposes the rebalance recommendation through two primary mechanisms: the Instance Metadata Service (IMDS) and Amazon EventBridge.

Polling the Instance Metadata Service

Applications or node agents running directly on the Spot Instance can poll the local Instance Metadata Service at regular intervals (for example, every 5 seconds). Using IMDSv2, you first retrieve a session token and then query the events/recommendations/rebalance endpoint:

TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/events/recommendations/rebalance

When a recommendation is active, the endpoint returns an HTTP 200 response containing a JSON payload with the noticeTime timestamp:

{
  "noticeTime": "2026-11-15T10:50:00Z"
}

If no recommendation is present, the endpoint returns an HTTP 404 status code.

Listening to Amazon EventBridge Events

For centralized orchestration or serverless automation, you can configure an Amazon EventBridge rule to capture the signal without running a polling agent on the instance. EventBridge emits an event with the following structure:

{
  "version": "0",
  "id": "12345678-1234-1234-1234-123456789012",
  "detail-type": "EC2 Instance Rebalance Recommendation",
  "source": "aws.ec2",
  "account": "111122223333",
  "time": "2026-11-15T10:50:00Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ec2:us-east-1b:instance/i-0abcdef1234567890"
  ],
  "detail": {
    "instance-id": "i-0abcdef1234567890"
  }
}

This event can trigger an AWS Lambda function, an AWS Systems Manager Run Command, or an AWS Step Functions workflow to begin draining the node and provisioning a replacement.

Native AWS Integrations That Handle the Signal Automatically

Before building custom logic to poll IMDS or process EventBridge events, evaluate whether your compute platform already handles the rebalance recommendation natively:

  • Amazon EC2 Auto Scaling: When you enable the Capacity Rebalancing feature on an Auto Scaling group, it automatically responds to the rebalance signal by launching a new Spot Instance from a diversified pool before terminating the at-risk instance.
  • Amazon EKS and Kubernetes: The open-source AWS Node Termination Handler (NTH) and Karpenter both support the rebalance recommendation. In Karpenter, enabling interruption handling causes it to cordon, drain, and replace nodes as soon as a rebalance signal is detected, while NTH can be configured to cordon and taint nodes proactively.
  • Amazon ECS: Managed Spot capacity providers and ECS container agents can be configured with Spot Instance draining to stop placing new tasks on an at-risk instance and safely migrate running tasks.
  • EC2 Fleet: Fleet configurations running in maintain mode can automatically launch replacement Spot capacity upon receiving a rebalance recommendation.

When to Ignore the Rebalance Signal

While the rebalance recommendation improves availability for slow-draining or slow-booting workloads, acting on it unconditionally can sometimes increase costs or cause unnecessary churn. Because the signal only indicates an elevated risk rather than a guaranteed interruption, immediately terminating an instance upon receiving the signal might discard a node that would have continued running for hours.

You should consider ignoring the rebalance recommendation—and relying solely on the two-minute interruption notice—when:

  1. Your workload is completely stateless and terminates cleanly in under 60 seconds.
  2. Your cluster maintains enough over-provisioned headroom to absorb the sudden loss of a node while a replacement boots.
  3. Your jobs are short-lived batch tasks where restarting a failed two-minute task is cheaper than preemptively migrating nodes on every risk warning.

For most production environments, the optimal strategy is a hybrid approach: upon receiving the rebalance recommendation, stop scheduling new work onto the instance and begin launching a replacement node in the background, but allow existing short-running tasks to continue until either they finish naturally or the two-minute interruption notice arrives.