How to Identify EC2 Instance vs. Volume EBS Bottlenecks?

Identifying whether an Amazon Elastic Block Store performance bottleneck is occurring at the Amazon Elastic Compute Cloud instance level or the individual volume level requires comparing aggregate instance metrics against individual storage volume metrics in Amazon CloudWatch. Every EC2 instance type enforces a maximum ceiling for total EBS bandwidth and input/output operations per second across all attached volumes, while each EBS volume enforces its own provisioned limits based on its volume type and configuration. By analyzing specific CloudWatch metrics for micro-bursting, throughput, IOPS, and queue length alongside operating system tools like iostat, administrators can quickly pinpoint the exact source of storage throttling and resolve latency issues.

Understanding the Two Layers of EBS Performance Limits

When an application reads from or writes to an Amazon EBS volume, the storage traffic must pass through two distinct performance gates:

1. The EC2 Instance Limit: EBS-optimized instances allocate dedicated network bandwidth between the EC2 instance and the EBS storage fleet. Each instance size has a baseline and burst (or maximum) limit for both IOPS and throughput (measured in megabytes per second). If you attach multiple high-performance volumes to a single instance, their combined traffic cannot exceed the ceiling of the instance itself.
2. The EBS Volume Limit: Each individual EBS volume (gp3, gp2, io2, io1, st1, sc1) has its own performance cap. For example, a standard gp3 volume provides a baseline of \(3,000\text{ IOPS}\) and \(125\text{ MB/s}\) of throughput unless scaled higher. Even if your EC2 instance supports \(40,000\text{ IOPS}\), a single default gp3 volume will throttle once demand exceeds \(3,000\text{ IOPS}\).

The effective performance for your workload is determined by the lower of the two limits:

\[\text{Effective IOPS} = \min\left(\text{Instance Max IOPS}, \sum_{i=1}^{n} \text{Volume}_i\text{ Max IOPS}\right)\]

\[\text{Effective Throughput} = \min\left(\text{Instance Max Throughput}, \sum_{i=1}^{n} \text{Volume}_i\text{ Max Throughput}\right)\]

How to Identify Instance-Level EBS Bottlenecks

An instance-level bottleneck occurs when the combined I/O demand of one or more attached EBS volumes exceeds the capacity of the EC2 instance's dedicated EBS connection.

1. Check Instance EBS Exceeded Metrics in CloudWatch

AWS provides two specific CloudWatch metrics under the AWS/EC2 namespace that directly measure instance-level EBS throttling (available on Nitro-based instances):

• EBSIOBalance% and EBSByteBalance% (for burstable EBS-optimized instances like t3, m5.large, or c5.large): Many smaller instance sizes can burst their EBS performance for up to 30 minutes at a time using a credit bucket. When EBSIOBalance% or EBSByteBalance% drops to \(0\%\), the instance falls back to its much lower baseline IOPS or throughput limit.
• EBSIOPSExceededCheck and EBSThroughputExceededCheck: Detailed CloudWatch metrics or AWS Compute Optimizer findings can reveal when aggregate demand hits the instance ceiling.

2. Calculate Aggregate Instance IOPS and Throughput

If you suspect the instance is hitting its maximum limit, sum the EBSReadOps and EBSWriteOps (or EBSReadBytes and EBSWriteBytes) metrics in the AWS/EC2 namespace and divide by the period in seconds:

\[\text{Instance IOPS} = \frac{\text{Sum}(\text{EBSReadOps}) + \text{Sum}(\text{EBSWriteOps})}{\text{Period in Seconds}}\]

\[\text{Instance Throughput (MiB/s)} = \frac{\text{Sum}(\text{EBSReadBytes}) + \text{Sum}(\text{EBSWriteBytes})}{\text{Period in Seconds} \times 1,048,576}\]

Compare these calculated values against the published maximum EBS performance specifications for your EC2 instance type in the AWS documentation. If your calculated values plateau right at the instance specification while application latency increases, the EC2 instance is the bottleneck.

How to Identify Volume-Level EBS Bottlenecks

A volume-level bottleneck occurs when a specific EBS volume reaches its maximum provisioned IOPS or throughput, even while the EC2 instance still has plenty of available EBS bandwidth.

1. Monitor Volume Performance Metrics in CloudWatch

Navigate to the AWS/EBS namespace in CloudWatch and inspect the following metrics for each individual VolumeId:

• VolumeReadOps and VolumeWriteOps: Divide the sum of these two metrics by the period in seconds to determine the actual IOPS consumed by the volume:

\[\text{Volume IOPS} = \frac{\text{Sum}(\text{VolumeReadOps}) + \text{Sum}(\text{VolumeWriteOps})}{\text{Period in Seconds}}\]

• VolumeReadBytes and VolumeWriteBytes: Divide the sum of these metrics by the period in seconds to determine the volume throughput:

\[\text{Volume Throughput (MiB/s)} = \frac{\text{Sum}(\text{VolumeReadBytes}) + \text{Sum}(\text{VolumeWriteBytes})}{\text{Period in Seconds} \times 1,048,576}\]

• VolumeQueueLength: This metric tracks the number of I/O requests waiting to be completed by the volume. If VolumeQueueLength spikes alongside high latency (VolumeTotalReadTime and VolumeTotalWriteTime), and the calculated IOPS or throughput matches the volume's provisioned limit, the individual volume is throttling.
• BurstBalance: If you are using legacy gp2, st1, or sc1 volumes, check the BurstBalance metric. If BurstBalance reaches \(0\%\), the volume has exhausted its I/O credits and is throttled down to its baseline performance.

2. Watch Out for Micro-Bursting

CloudWatch aggregates basic EBS metrics in 1-minute or 5-minute intervals. However, EBS enforces performance limits on a 1-second basis. A volume might spike to \(10,000\text{ IOPS}\) for 5 seconds and sit idle for 55 seconds. Over a 1-minute CloudWatch period, the average will report only \(833\text{ IOPS}\), hiding the fact that the volume was severely throttled during those 5 active seconds.

To detect micro-bursting on Nitro-based EC2 instances, check the high-resolution VolumeConsumedReadWriteOps metric or use Amazon EBS Detailed Performance Statistics via ethtool or NVMe driver statistics inside the guest operating system.

Diagnosing Bottlenecks Inside the Linux Operating System

You can verify CloudWatch findings in real time with the iostat utility (part of the sysstat package) and NVMe CLI tools on Linux instances.

Run iostat with a 1-second interval to catch micro-bursting and compare individual disk behavior:

iostat -xz 1

Examine the following columns in the output:

• r/s and w/s: The read and write operations per second per device.
• rkB/s and wkB/s: The kilobytes read and written per second.
• r_await and w_await: The average time (in milliseconds) for I/O requests to be serviced.
• aqu-sz: The average queue length of requests.

Comparing Multi-Volume Behavior

Suppose an EC2 instance has three attached gp3 volumes (nvme1n1, nvme2n1, and nvme3n1):

• Scenario A (Volume Bottleneck): nvme1n1 shows aqu-sz > \(4\) and r_await > \(15\text{ ms}\) while running right at \(3,000\text{ IOPS}\), whereas nvme2n1 and nvme3n1 show sub-millisecond await times and low utilization. This confirms that nvme1n1 is bottlenecked at the volume level.
• Scenario B (Instance Bottleneck): All three volumes show elevated await times and growing queue lengths simultaneously whenever their combined throughput hits \(600\text{ MB/s}\) (for example, on an m5.xlarge after burst credits expire), even though each volume is provisioned for \(500\text{ MB/s}\). This confirms an instance-level bottleneck.

Checking Nitro NVMe Driver Performance Statistics

On modern Nitro-based EC2 instances, Amazon EBS exposes detailed hardware-level throttling counters directly to the guest OS through the NVMe driver. You can inspect whether the instance or the volume exceeded its limits with the nvme command-line tool:

sudo nvme amzn stats /dev/nvme1n1

Look for the following counters in the output:

• ebsnvme-volume_iops_exceeded: Total seconds within the measurement window where the volume exceeded its provisioned IOPS limit.
• ebsnvme-volume_throughput_exceeded: Total seconds where the volume exceeded its provisioned throughput limit.
• ebsnvme-instance_iops_exceeded: Total seconds where the EC2 instance's aggregate EBS IOPS limit was exceeded.
• ebsnvme-instance_throughput_exceeded: Total seconds where the EC2 instance's aggregate EBS throughput limit was exceeded.

If the instance_*_exceeded counters are incrementing, your EC2 instance is the bottleneck. If the volume_*_exceeded counters are incrementing while the instance counters remain at zero, the individual EBS volume is the bottleneck.

Summary Comparison Matrix