What File Systems Support Amazon EBS Multi-Attach Writes?

Amazon Elastic Block Store Multi-Attach allows a single provisioned IOPS SSD volume to be attached to up to sixteen Amazon EC2 instances in the same Availability Zone simultaneously. Standard local file systems such as XFS, ext4, and NTFS are not designed for shared block storage and will rapidly cause severe data corruption if multiple instances write to the volume at the same time. To safely coordinate concurrent writes across multiple EC2 instances using EBS Multi-Attach, you must use a cluster-aware file system such as Red Hat Global File System 2 or Oracle Cluster File System version 2, which implement distributed locking mechanisms and cache coherency protocols to keep metadata and file blocks synchronized across all attached nodes.

Why Standard File Systems Fail with EBS Multi-Attach

Amazon EBS Multi-Attach operates strictly at the raw block storage layer. When you attach an io1 or io2 volume with Multi-Attach enabled to \(N\) instances (where \(1 \le N \le 16\)), the underlying EBS service exposes the exact same logical block addresses to each attached EC2 instance without any built-in file-level or block-level write coordination.

Standard local file systems—including ext4, XFS, Btrfs, and Windows NTFS—assume exclusive control over the block device. They rely on local in-memory caches for file system metadata, allocation bitmaps, directory structures, and journal entries:

1. In-Memory Metadata Caching: Instance A reads the block allocation bitmap into its local RAM and allocates block \(B_k\) for a new file. Before or even after Instance A flushes this change to disk, Instance B reads its own cached version of the allocation bitmap, sees block \(B_k\) as free, and overwrites it with completely different data.
2. Uncoordinated Journaling: Each instance maintains or replays a journal assuming no other writer exists. Concurrent updates to the superblock and inode tables result in immediate structural corruption.
3. Buffer Cache Incoherency: Even if only one instance writes and other instances mount the volume read-only, standard file systems do not automatically invalidate their local page caches when the underlying disk blocks change externally. Readers see stale or partially updated metadata.

Required Cluster-Aware File Systems for Concurrent Writes

To enable safe, concurrent read-write operations across \(N\) attached EC2 instances, you must format the volume with a shared-disk, cluster-aware file system. These file systems embed a Distributed Lock Manager (DLM) that coordinates access across all nodes over the network before committing reads or writes to the block device.

1. GFS2 (Global File System 2)

GFS2 is the most widely deployed native Linux cluster file system for EBS Multi-Attach workloads. Available in Red Hat Enterprise Linux (RHEL) through the Resilient Storage Add-On, Rocky Linux, AlmaLinux, and Ubuntu, GFS2 allows all nodes in a Pacemaker and Corosync cluster to mount the same EBS volume concurrently in read-write mode.

• Locking Mechanism: Uses the Linux kernel's Distributed Lock Manager (dlm) over TCP/IP or SCTP between EC2 instances.
• Journaling: Allocates a dedicated journal on the shared disk for each participating node. If a cluster has \(N\) nodes, the file system creates \(N\) journals (journal0 through journalN-1), allowing crashed nodes to have their journals replayed by surviving nodes after fencing.
• Cache Coherency: Implemented via glocks (GFS2 locks), which synchronize both DLM lock states and local VFS page cache invalidation across nodes.

2. OCFS2 (Oracle Cluster File System Version 2)

OCFS2 is a general-purpose, extent-based shared-disk file system integrated into the mainline Linux kernel and heavily supported on Oracle Linux, SUSE Linux Enterprise Server (SLES), and Ubuntu.

• Locking Mechanism: Includes a built-in cluster stack (o2cb) with an integrated DLM, or can integrate with Pacemaker and Corosync (pcmk).
• Use Cases: Originally designed for Oracle Real Application Clusters (RAC) datafiles, archive logs, and shared configuration files, it also serves general POSIX workloads such as shared web roots, media processing pipelines, and high-availability application state.
• Block Allocation: Uses global bitmaps paired with node-local allocation windows to minimize lock contention when multiple EC2 instances create files simultaneously.

3. VMware VMFS and Veritas InfoScale (Cluster File System)

In specialized enterprise environments on AWS:

• Veritas Cluster File System (VxFS / InfoScale): Enterprise commercial shared file system that provides high-performance distributed locking, fast failover, and multi-node concurrent read/write support on EC2.
• Application-Managed Raw Block Storage: Databases such as Oracle RAC (using Oracle Automatic Storage Management, or ASM) bypass traditional OS file systems entirely, managing raw block locking, striping, and cache fusion directly within the database cluster layer.

Architectural Requirements for Running a Cluster File System on EC2

Formatting an EBS Multi-Attach volume with GFS2 or OCFS2 is only one part of a safe deployment. A complete cluster file system architecture requires three core components:

Distributed Lock Management Overhead

Every metadata modification—and any overlapping data block write—requires inter-node lock negotiation over the VPC network. In an \(N\)-node cluster, the worst-case lock coordination message traffic scales with the number of active nodes:

\[\text{Communication Paths} = \frac{N(N - 1)}{2}\]

To keep lock acquisition latency low, all attached EC2 instances must reside in the same Availability Zone (a hard requirement of EBS Multi-Attach) and should ideally be placed in a Cluster Placement Group for low-latency, non-blocking network communication.

Node Fencing (STONITH)

If an EC2 instance loses network connectivity to the Corosync/DLM cluster ring but retains connectivity to the EBS storage fabric, it might continue writing buffered I/O to the EBS volume while surviving nodes assume it is dead and replay its journal. To prevent split-brain corruption, the cluster must enforce fencing (Shoot The Other Node In The Head, or STONITH).

Because Amazon EBS volumes (io1 and io2) do not support SCSI-3 Persistent Reservations (SCSI-3 PR) for I/O fencing, cluster file systems on EC2 must use API-level power fencing:

• fence_aws Fencing Agent: Configured in Pacemaker to call the AWS EC2 API (StopInstances or RebootInstances) via an IAM role whenever a node becomes unresponsive, guaranteeing that all in-flight I/O from the failed instance is terminated before any surviving node replays its journal.

EBS Volume Constraints to Keep in Mind

When planning a cluster file system on EBS Multi-Attach, account for these platform constraints:

• Supported Volume Types: Only Provisioned IOPS SSD (io1 and io2) volumes support Multi-Attach. General Purpose SSD (gp2, gp3) and magnetic volumes do not support it.
• Instance Support: Attachable only to Nitro-based EC2 instances within a single Availability Zone.
• Aggregate Throughput and IOPS Limits: Total volume performance is shared across all attached instances. If a volume is provisioned for \(I_{\text{max}}\) IOPS and \(N\) instances perform equal concurrent workloads, the average IOPS available per instance is:

\[I_{\text{instance}} = \frac{I_{\text{max}}}{N}\]

Furthermore, each individual EC2 instance remains bound by its own instance-level EBS bandwidth and IOPS limits.

Comparison of Storage Options for Concurrent EC2 Access

When to Use Managed Shared File Systems Instead

Setting up GFS2 or OCFS2 on EBS Multi-Attach requires configuring Corosync, Pacemaker, DLM, fence_aws IAM permissions, and custom kernel modules. Unless your application specifically requires sub-millisecond raw block latency or direct block device semantics within a single Availability Zone, AWS recommends using a managed shared file system such as Amazon EFS or Amazon FSx (for Lustre, NetApp ONTAP, or OpenZFS), which handle concurrent write locks natively across multiple Availability Zones without cluster management overhead.