How Does Amazon EFS Handle Concurrent Cross-AZ Writes?

When multiple Amazon EC2 instances across different Availability Zones write concurrently to the same standard Amazon Elastic File System, the service maintains data integrity using the Network File System version 4 protocol. Standard EFS replicates data synchronously across multiple Availability Zones before acknowledging any write operation, ensuring durability and immediate cross-zone visibility. Concurrent modifications are governed by close-to-open consistency semantics and POSIX-compliant byte-range locking, which allow distributed applications to coordinate file access safely without data corruption, though uncoordinated writes to the exact same byte range will result in the last write winning.

Synchronous Multi-AZ Replication and Durability

A standard Amazon EFS file system stores data and metadata redundantly across multiple Availability Zones within an AWS Region. When an EC2 instance writes data to EFS through a local mount target, the write operation is not immediately considered complete when it reaches the initial storage node. Instead, EFS synchronously replicates the data across multiple Availability Zones.

If an instance writes a data block of size \(B\), the total round-trip latency \(L_{\text{write}}\) includes the local network transit time \(T_{\text{local}}\) to the mount target and the inter-AZ replication time \(T_{\text{sync}}\):

\[L_{\text{write}} = T_{\text{local}} + T_{\text{sync}} + T_{\text{commit}}\]

Only after the data is durably committed across multiple Availability Zones does EFS return a success acknowledgment to the NFS client on the EC2 instance. This architecture guarantees that if an entire Availability Zone becomes unavailable, no acknowledged writes are lost, and instances in surviving Availability Zones continue operating against the exact same state of the file system.

Close-to-Open Consistency in NFSv4

Amazon EFS uses the NFSv4.1 (and NFSv4.0) protocol, which implements a consistency model known as close-to-open consistency. Because NFS clients on EC2 instances cache file data and attributes locally in memory to improve performance, concurrent access across different instances requires explicit synchronization boundaries.

Under close-to-open consistency:

When an application on Instance A modifies a file and calls the close() system call, the NFS client flushes all dirty cached data blocks to the EFS service and waits for durable persistence.
When an application on Instance B in another Availability Zone subsequently calls open() on that same file, its NFS client validates its local cache against the EFS metadata state.
Because the file modification time and change attribute have been updated by Instance A, Instance B invalidates its stale local page cache and fetches the updated data directly from EFS.

This guarantees that any file closed by one instance is immediately and completely visible to any other instance that opens the file afterward, regardless of which Availability Zone the instances reside in.

Uncoordinated Concurrent Writes to the Same File

When multiple EC2 instances open the same file simultaneously and write to it without closing the file or using file locks, the outcome depends on which byte ranges are being modified and how data flushing is handled.

Disjoint Byte Ranges

If Instance A writes to bytes \(0\) through \(4095\) and Instance B writes to bytes \(4096\) through \(8191\), both updates can succeed without overwriting each other, provided the writes align cleanly with page boundaries or are flushed explicitly using fsync(). When both instances eventually close the file or call fsync(), EFS merges the changes into the underlying file object.

Overlapping Byte Ranges

If multiple instances write to the exact same byte range concurrently without coordination, a race condition occurs. EFS processes incoming RPC write requests sequentially as they arrive at the storage layer. In this scenario, the file system enforces last-writer-wins semantics at the block or byte range level. Because NFS clients buffer writes locally in page caches before flushing them asynchronously, the order in which application write() calls occur in wall-clock time may not match the order in which flush requests reach EFS. Without explicit synchronization, overlapping concurrent writes from multiple instances can result in interleaved data blocks and corrupted file structures.

Appending Data Across Instances

Standard POSIX O_APPEND semantics are not strictly atomic across distributed NFS clients. When an instance opens a file in append mode, the NFS client determines the end-of-file offset based on its cached file attributes. If two instances in different Availability Zones append to a shared log file simultaneously, both clients might read the same current file size \(S\) and issue writes starting at offset \(S\), causing one instance's log entry to overwrite the other's.

Coordinating Access with File Locking and Direct I/O

To prevent race conditions and data corruption during concurrent cross-AZ writes, applications rely on three primary mechanisms supported by Amazon EFS:

POSIX Advisory Locking (fcntl and flock): NFSv4 integrates stateful locking directly into the protocol. An application on an EC2 instance can request an exclusive write lock or a shared read lock on a whole file or a specific byte range \([x, y]\). EFS maintains the authoritative lock state centrally. When a lock is acquired or released, the client automatically flushes dirty cache pages and invalidates stale reads for that byte range, guaranteeing serialized, strongly consistent updates across all Availability Zones.
Explicit Cache Flushing (fsync): Calling fsync() or fdatasync() forces the local NFS client on the EC2 instance to immediately transmit all buffered writes to EFS and wait for multi-AZ persistence without closing the file descriptor.
Direct I/O (O_DIRECT): Opening files with the O_DIRECT flag bypasses the operating system page cache on the EC2 instance entirely. Every read and write operation goes directly over the network to the EFS mount target. When combined with byte-range locking or application-level offset coordination, Direct I/O eliminates cache coherency delays across instances.
Atomic Rename Pattern: For configuration files or shared artifacts, instances often write new content to a temporary, uniquely named file in the same directory, flush it with fsync(), close it, and execute a POSIX rename() operation over the target filename. In EFS, rename() is atomic, ensuring that concurrent readers across any Availability Zone see either the complete old file or the complete new file, never a partially written file.

Performance and Throughput Behavior Across Availability Zones

When hundreds or thousands of EC2 instances write concurrently to a single EFS file system, throughput and IOPS scale elastically, but contended resources behave differently than uncontended ones:

Aggregate Bandwidth: Each Availability Zone uses its own EFS mount target. Aggregate throughput is shared across the entire file system, meaning concurrent writes from multiple Availability Zones draw from the same file system throughput pool (whether configured in Elastic, Provisioned, or Bursting throughput mode).
Metadata Contention: Writing to separate files spread across separate directories scales horizontally with minimal contention. Conversely, if many instances concurrently create, delete, or append to the exact same file or within a single directory, EFS must serialize the metadata mutations and lock state transitions, which increases per-operation latency.
Cross-AZ Mount Target Traffic: If an EC2 instance mounts an EFS mount target located in its own Availability Zone, there are no EC2 inter-AZ data transfer charges for the EFS traffic, as the synchronous multi-AZ replication happens internally within the EFS service boundary. Mounting a mount target located in a different Availability Zone incurs standard cross-AZ data transfer fees and adds network latency.