How to Test EC2 Spot Resilience Using AWS FIS?

Testing application resilience against unplanned Amazon EC2 Spot Instance terminations using AWS Fault Injection Service involves creating controlled chaos engineering experiments that simulate the exact two-minute interruption notice and subsequent instance reclamation emitted by AWS. By configuring an AWS Fault Injection Service experiment template targeting specific Spot Instances with the send-spot-instance-interruptions action, engineering teams can validate that their workload orchestration layers, such as Amazon EKS, Auto Scaling groups, or custom instance termination handlers, gracefully drain connections, checkpoint state, and provision replacement capacity without degrading service availability.

Understanding Amazon EC2 Spot Interruptions and AWS FIS

Amazon EC2 Spot Instances allow organizations to utilize spare AWS compute capacity at substantial discounts compared to On-Demand pricing. However, the fundamental trade-off of using Spot capacity is that Amazon EC2 can reclaim these instances at any time when capacity is needed back or when the Spot price exceeds the maximum configured price. When AWS reclaims a Spot Instance, it provides a two-minute warning known as a Spot Instance interruption notice before terminating, stopping, or hibernating the instance.

Historically, testing how an application responds to this two-minute window was difficult because teams could not predict or force genuine capacity reclamations on demand. Calling a standard TerminateInstances API does not trigger the same metadata events or Amazon EventBridge notifications as a real Spot reclamation. AWS Fault Injection Service solves this problem by providing a native fault action called aws:ec2:send-spot-instance-interruptions. When triggered, this action initiates the authentic Spot interruption workflow, delivering the two-minute warning via Both the Instance Metadata Service and Amazon EventBridge, followed by the actual termination or stop action.

Prerequisites for Running Spot Interruption Experiments

Before executing a fault injection experiment on your EC2 Spot fleet, several architectural and security prerequisites must be in place to ensure the test is both realistic and safe.

Target Spot Instances and Tagging Strategy

The aws:ec2:send-spot-instance-interruptions action only works on running EC2 Spot Instances. It cannot be applied to On-Demand or Reserved Instances. To prevent accidental disruption of non-test workloads, you should apply dedicated resource tags to your target Spot Instances, such as Environment=ChaosTesting or FIS-Target=True. If your Spot Instances are managed by an EC2 Auto Scaling group or an Amazon EKS Managed Node Group, ensure that tag propagation to launched instances is enabled.

Interruption Handling Software

To observe a meaningful result, your instances or cluster must be running software designed to detect and react to Spot interruption signals. Common mechanisms include:

  • AWS Node Termination Handler (NTH): Deployed as a DaemonSet on Kubernetes clusters (such as Amazon EKS) to poll the Instance Metadata Service or listen to EventBridge events via Amazon SQS, cordoning the node and draining pods when an interruption occurs.
  • Amazon EC2 Auto Scaling Capacity Rebalancing: A feature that proactively launches a replacement Spot Instance when an existing instance emits a rebalance recommendation or interruption notice.
  • Custom Polling Scripts or EventBridge Rules: Application-specific daemons that query http://169.254.169.254/latest/meta-data/spot/instance-action every five seconds or consume EC2 Spot Instance Interruption Warning events from EventBridge to flush caches, close database connections, or upload logs.

AWS IAM Role for Fault Injection Service

AWS Fault Injection Service requires an IAM role with permissions to inspect target EC2 instances and inject the Spot interruption fault. First, create a trust policy that allows fis.amazonaws.com to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "fis.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Next, attach a permissions policy granting the role authority to describe instances and send Spot interruptions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDescribeInstancesForTargeting",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "ec2:DescribeInstances"
      ]
    },
    {
      "Sid": "AllowSpotInstanceInterruption",
      "Effect": "Allow",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Action": [
        "ec2:SendSpotInstanceInterruptions"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/FIS-Target": "True"
        }
      }
    }
  ]
}

Designing the Resilience Hypothesis and Steady-State Metrics

Chaos engineering requires a well-defined hypothesis before injecting faults. When testing Spot resilience, your hypothesis generally states that if a percentage of Spot Instances receive an interruption notice, the application will maintain its service-level objectives without dropping in-flight client requests.

You can quantify your availability and error budget during the experiment using standard reliability formulas. For instance, request success rate \(R_{\text{success}}\) over the experiment window is defined as:

\[R_{\text{success}} = \left( 1 - \frac{N_{\text{5xx}}}{N_{\text{total}}} \right) \times 100\]

where \(N_{\text{5xx}}\) represents the count of server-side HTTP error responses and \(N_{\text{total}}\) represents the total requests processed by the load balancer.

Furthermore, to ensure that your graceful shutdown logic finishes before the hard termination deadline, the total drain time \(T_{\text{drain}}\) must satisfy the inequality:

\[T_{\text{detect}} + T_{\text{deregister}} + T_{\text{inflight}} < 120\text{ seconds}\]

where \(T_{\text{detect}}\) is the latency to detect the notice, \(T_{\text{deregister}}\) is the time required to stop new traffic routing at the load balancer, and \(T_{\text{inflight}}\) is the time taken to complete active transactions.

To enforce safety during the test, create an Amazon CloudWatch Alarm that monitors \(R_{\text{success}}\) or target group healthy host counts. This alarm will serve as the Stop Condition in AWS FIS, automatically aborting the experiment if application health degrades beyond acceptable thresholds.

Step-by-Step Guide to Creating the AWS FIS Experiment Template

You can configure the experiment template using the AWS Management Console, AWS CloudFormation, Terraform, or the AWS CLI. Below is the complete workflow for building and executing the template using the AWS CLI.

1. Define the Experiment Template JSON

Create a file named spot-interruption-template.json. This configuration defines the target selection criteria, the aws:ec2:send-spot-instance-interruptions action, and the CloudWatch stop condition.

{
  "description": "Simulate EC2 Spot Instance interruptions to verify graceful draining and auto-recovery",
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:HighALB5xxErrorRate"
    }
  ],
  "targets": {
    "SpotInstancesTarget": {
      "resourceType": "aws:ec2:spot-instance",
      "resourceTags": {
        "FIS-Target": "True"
      },
      "filters": [
        {
          "path": "State.Name",
          "values": [
            "running"
          ]
        }
      ],
      "selectionMode": "PERCENT(50)"
    }
  },
  "actions": {
    "InterruptSpotInstances": {
      "actionId": "aws:ec2:send-spot-instance-interruptions",
      "description": "Trigger two-minute warning and terminate selected Spot Instances",
      "parameters": {
        "durationBeforeInterruption": "PT2M"
      },
      "targets": {
        "SpotInstances": "SpotInstancesTarget"
      }
    }
  },
  "roleArn": "arn:aws:iam::123456789012:role/FISSpotExperimentRole",
  "tags": {
    "Name": "Spot-Resilience-Test"
  }
}

2. Understanding Key Template Parameters

  • resourceType: Must be set to aws:ec2:spot-instance rather than the generic aws:ec2:instance type.
  • selectionMode: Controls the blast radius of the experiment. You can specify ALL to interrupt every matching instance, COUNT(n) to interrupt a fixed number of instances (e.g., COUNT(2)), or PERCENT(n) to interrupt a specific percentage of matching instances (e.g., PERCENT(50)).
  • durationBeforeInterruption: Specified in ISO 8601 duration format. It defines how long AWS FIS waits between initiating the action and triggering the actual reclamation. For example, setting PT2M means the interruption notice is sent immediately when the action starts, and the instance is terminated two minutes later. If you set PT5M, AWS FIS waits three minutes before emitting the two-minute interruption notice, terminating the instance at the five-minute mark. The minimum supported value is PT2M.

3. Create the Template via AWS CLI

Run the following command to register the experiment template in AWS Fault Injection Service:

aws fis create-experiment-template \
    --cli-input-json file://spot-interruption-template.json \
    --region us-east-1

Record the id field returned in the JSON output (for example, EXT1234567890abcdef), as you will need it to start the experiment.

Executing the Experiment and Observing System Behavior

Before starting the experiment, ensure that realistic synthetic traffic or load testing is running against your application so you can accurately measure whether in-flight connections are dropped.

Start the experiment using the template ID:

aws fis start-experiment \
    --experiment-template-id EXT1234567890abcdef \
    --region us-east-1

Once the experiment transitions to the running state, the following sequence of events takes place across your AWS environment:

  1. Emission of Rebalance and Interruption Signals: At the appropriate timestamp, AWS emits an EC2 Instance Rebalance Recommendation event and an EC2 Spot Instance Interruption Warning event to Amazon EventBridge. Simultaneously, the Instance Metadata Service (IMDSv2) on the targeted Spot Instances populates the /latest/meta-data/spot/instance-action and /latest/meta-data/events/recommendations/rebalance endpoints.
  2. Detection by Termination Handlers: If you are verifying an instance-level script or Kubernetes node handler, you can inspect what the instance sees during the two-minute window by querying IMDSv2 directly from the instance:
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -s)
curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/spot/instance-action

This returns a JSON payload detailing the action (terminate, stop, or hibernate) and the exact UTC timestamp when the action will be executed:

{
  "action": "terminate",
  "time": "2026-11-15T11:02:00Z"
}
  1. Graceful Workload Draining: Your termination handler or orchestration agent intercepts the signal. On an Application Load Balancer (ALB) target group, the instance enters the draining state. Ensure your target group's Deregistration Delay (connection draining timeout) is configured to a value lower than 120 seconds (such as 60 or 90 seconds) so existing requests finish before the underlying EC2 instance is powered off.
  2. Instance Termination and Replacement: Exactly two minutes after the warning notice is issued, EC2 terminates the targeted Spot Instances. Concurrently, your Auto Scaling group or Karpenter node provisioner detects the capacity drop (or acts early on the rebalance recommendation) and provisions replacement instances across diversified Spot pools.

Validating Results and Remediating Common Bottlenecks

After the experiment reaches the completed state, analyze your observability dashboards, CloudWatch logs, and load balancer metrics to verify your resilience hypothesis:

  • Zero HTTP 502/504 Gateway Errors: Check your Application Load Balancer or API Gateway metrics. If HTTPCode_ELB_502_Count or HTTPCode_ELB_504_Count spiked during the two-minute window, your application continued accepting new requests after the interruption notice arrived, or its keep-alive timeout conflicted with the load balancer. Configure your web server's graceful shutdown hook to immediately fail its health check endpoint upon receiving a SIGTERM or Spot interruption notice, while keeping the main listener open briefly to finish active requests.
  • Pod Eviction and Scheduling Delays in Kubernetes: In Amazon EKS, if pods were forcefully killed (SIGKILL) when the node vanished, examine the terminationGracePeriodSeconds in your Pod specifications. If a pod is configured with a 180-second grace period, it will exceed the 120-second Spot reclamation window. Restrict terminationGracePeriodSeconds on Spot-eligible workloads to 90 seconds or less, and implement preStop lifecycle hooks to handle connection cleanup cleanly.
  • Slow Replacement Provisioning: If your remaining capacity became overwhelmed after the targeted Spot Instances terminated, enable Capacity Rebalancing on your Auto Scaling group or use Karpenter with interruption queue handling enabled. Both mechanisms begin launching replacement nodes as soon as the interruption notice is published, overlapping the boot time of the new node with the two-minute drain window of the terminating node.