Why Run Apache Cassandra or ScyllaDB on AWS I4i Instances?

Distributed NoSQL databases like Apache Cassandra and ScyllaDB are regularly deployed on Amazon EC2 I4i instances because these instances deliver ultra-low latency local NVMe storage, high random I/O throughput, and balanced compute-to-memory ratios that directly complement log-structured merge-tree storage engines. By pairing third-generation AWS Nitro SSDs with high-frequency Intel Xeon processors and enhanced networking, I4i instances eliminate storage bottlenecks during heavy write flushing, background compactions, and uncached point reads. This combination ensures predictable sub-millisecond tail latencies and lowers the total cost of ownership by reducing the total number of cluster nodes required to handle massive datasets.

Understanding the LSM-Tree Storage Architecture

To understand why hardware selection is critical for Apache Cassandra and ScyllaDB, it is necessary to examine how they store and retrieve data. Both databases rely on a Log-Structured Merge-tree (LSM-tree) storage engine rather than traditional B-trees used by relational databases.

When a write operation arrives at a node, the database performs two actions simultaneously:

1. It appends the write sequentially to an on-disk commit log for durability.
2. It writes the data into an in-memory structure called a Memtable.

Once the Memtable reaches its capacity limit, the database flushes its contents sequentially to disk as an immutable Sorted String Table (SSTable). Because SSTables are immutable, updates and deletes do not overwrite existing records in place; instead, they write new versions of the data or special markers called tombstones into newer SSTables.

Over time, a single partition key may have fragments scattered across multiple SSTables on disk. This creates two distinct hardware challenges:

• Compaction I/O: The database must continuously run background compaction processes that read multiple SSTables, merge the data, discard obsolete versions and expired tombstones, and write out new consolidated SSTables.
• Read Amplification: When a read request misses the in-memory caches, the node must check bloom filters and potentially read from several SSTables concurrently to assemble the complete record.

Consequently, an LSM-tree database demands storage hardware capable of sustaining massive sequential write bandwidth for commit logs and flushes, alongside ultra-fast random read I/O and high parallel throughput to prevent background compactions from starving client requests.

Key Hardware Features of Amazon EC2 I4i Instances

Amazon EC2 I4i instances are storage-optimized virtual machines purpose-built for I/O-intensive workloads. They improve significantly upon the previous I3 and I3en generations across several architectural dimensions.

AWS Nitro SSDs and Local NVMe Storage

The defining feature of the I4i family is its local NVMe storage powered by custom AWS Nitro SSDs. Unlike network-attached storage such as Amazon EBS, which traverses the network fabric and incurs serialization overhead, Nitro SSDs are physically attached to the host server.

Nitro SSDs are engineered specifically to minimize I/O latency variability. Compared to third-party commercial SSDs used in older instance generations, AWS Nitro SSDs deliver up to 60% lower storage I/O latency and up to 75% reduced latency variability. In distributed databases where a client request often waits on the slowest responding replica, eliminating disk latency spikes is essential for maintaining strict \(p_{99}\) and \(p_{99.9}\) service-level agreements (SLAs).

Compute Performance with Intel Xeon Ice Lake Processors

I4i instances are powered by 3rd Generation Intel Xeon Scalable processors (code-named Ice Lake) running at an all-core turbo frequency of 3.5 GHz.

• For Apache Cassandra: Running on the Java Virtual Machine (JVM), Cassandra benefits from higher single-core clock speeds and improved instructions per cycle (IPC) to execute garbage collection pauses faster, serialize/deserialize network payloads, and compress SSTable blocks.
• For ScyllaDB: Built in C++ on the Seastar asynchronous framework, ScyllaDB uses a shared-nothing, thread-per-core architecture. Each vCPU is pinned to a specific shard with its own memory and I/O queues. High-frequency Ice Lake cores allow each shard to process ratusan of thousands of operations per second without lock contention.

Memory-to-vCPU and Storage Ratios

Across all I4i sizes—from i4i.large up to i4i.32xlarge and i4i.metal—AWS maintains a consistent architectural ratio:

• 8 GiB of DDR4 memory per vCPU
• Approximately 234 GB of NVMe SSD storage per vCPU (up to 30 TB total on the largest instances)

This 8:1 memory-to-vCPU ratio is widely considered the sweet spot for NoSQL workloads. It provides ample RAM for Cassandra's off-heap row caches and OS page cache, or ScyllaDB's unified cache, ensuring that hot datasets are served directly from memory while keeping the per-node storage footprint small enough that streaming and repair operations remain manageable.

Enhanced Networking and EBS Bandwidth

In a ring or token-aware topology, nodes constantly communicate via gossip protocols, replicate mutations across availability zones, and stream data during topology changes. I4i instances provide up to 75 Gbps of network bandwidth with Elastic Fabric Adapter (EFA) and Enhanced Networking (ENA) support, preventing network saturation during heavy cross-rack replication or node rebuilds.

Why Local NVMe Outperforms Amazon EBS for Cassandra and ScyllaDB

A common architectural question is why teams choose ephemeral local NVMe drives on I4i instances rather than durable Amazon Elastic Block Store (EBS) volumes like gp3 or io2.

Built-In Database Redundancy vs. Storage-Layer Redundancy

Amazon EBS replicates data within an Availability Zone behind the scenes to protect against drive failures. However, Cassandra and ScyllaDB are already architected as shared-nothing distributed systems that handle replication natively at the application layer. Typically, a cluster is configured with a Replication Factor (\(RF\)) of 3 across three distinct Availability Zones.

Using EBS beneath Cassandra or ScyllaDB introduces redundant replication: every write is replicated three times across the cluster by the database, and each of those three writes is replicated again by EBS. By using the ephemeral local NVMe drives on I4i instances, architects eliminate the EBS replication tax, paying only for raw hardware speed while relying on the database's native replication and repair mechanisms for durability.

Overcoming IOPS Limits and Cost Bottlenecks

ScyllaDB in particular is designed to saturate modern NVMe drives, capable of driving ratusan of thousands of IOPS per node. Provisioning equivalent IOPS and throughput on Amazon EBS io2 Block Express volumes becomes prohibitively expensive due to per-IOPS provisioning charges, and still introduces sub-millisecond network round-trip overhead. The local NVMe drives on I4i instances include millions of IOPS natively within the base hourly instance price.

Mitigating Compaction Stalls

When an LSM-tree database performs major or size-tiered compactions, disk read and write throughput spikes dramatically. On network-attached storage with strict burst buckets or throughput caps, compaction easily exhausts available bandwidth, causing front-end read and write latencies to climb. The dedicated multi-gigabyte-per-second throughput of local Nitro SSDs allows I4i instances to absorb simultaneous compaction reads, compaction writes, and live client traffic without queuing delays.

Optimizing Consistency and Tail Latency

In Dynamo-style distributed databases, tunable consistency determines how many replicas must acknowledge an operation before it succeeds. Most production deployments use LOCAL_QUORUM for both reads and writes. The number of nodes required to satisfy a quorum is defined by:

\[Q = \left\lfloor \frac{RF}{2} \right\rfloor + 1\]

When \(RF = 3\), a LOCAL_QUORUM operation requires responses from at least \(Q = 2\) replicas. This means the perceived latency of any client request is dictated by the second-fastest replica (or the slowest of the two fastest replicas).

If nodes run on storage with erratic background garbage collection or noisy-neighbor storage contention, individual disks experience frequent latency spikes. When multiple replicas experience simultaneous micro-stalls, the cluster's overall \(p_{99}\) latency degrades rapidly. Because AWS Nitro SSDs offload storage management tasks to dedicated Nitro hardware controllers and minimize wear-leveling jitter, I4i instances keep individual replica response times tightly bounded, directly translating into flat \(p_{99}\) latency curves at the coordinator level.

Security and Operational Advantages

Beyond raw performance, I4i instances offer operational improvements tailored for large-scale database clusters:

• Always-On Hardware Encryption: All data written to local NVMe storage on I4i instances is automatically encrypted using XTS-AES-256 block cipher implemented directly in the Nitro hardware module. For compliance-heavy Cassandra and ScyllaDB deployments, this provides encryption at rest with zero CPU overhead or latency penalty.
• Torn-Write Prevention: I4i instances support torn-write prevention, allowing databases to write larger blocks atomically without the risk of partial page writes during unexpected power or system failures, reducing the overhead of double-write buffers or excessive checksumming.
• Bare Metal Options (i4i.metal): For organizations running ScyllaDB that want to bypass hypervisor virtualization entirely for direct hardware access to CPU caches and NVMe queues, i4i.metal provides 128 vCPUs, 1,024 GiB of memory, and 30 TB of local NVMe storage.

Summary of Workload Fit