How Do Lustre and GPFS Stripe Large Scientific Datasets?

Parallel file systems such as Lustre and GPFS enable high-performance computing clusters to read and write massive scientific datasets simultaneously across hundreds of storage servers. By dividing large files into fixed-size blocks and distributing those blocks across multiple independent storage targets, these systems eliminate single-server bottlenecks and aggregate the bandwidth of the entire storage network.

Core Architecture of Parallel Striping

In classical computing environments, a single storage server cannot saturate the aggregate network bandwidth of thousands of compute nodes. Parallel file systems solve this by separating metadata operations from bulk data transfer. Metadata servers (MDS) manage file names, permissions, and directory structures, while data servers handle the actual storage blocks.

When a client application writes a large dataset, the file is partitioned into chunks of a defined stripe size, denoted as \(S\). If a file has total size \(F\), the number of stripes \(N\) is given by:

\[N = \left\lceil \frac{F}{S} \right\rceil\]

These chunks are distributed in a round-robin fashion across a set of storage targets, known as the stripe count \(C\). Because each target resides on a separate physical disk array or storage server, multiple compute nodes can read or write different portions of the same file concurrently without waiting for a single disk controller.

Striping in Lustre

Lustre uses Object Storage Servers (OSS) that manage one or more Object Storage Targets (OSTs). Each OST represents a dedicated storage volume. When a file is created in Lustre, the Metadata Server (MDS) assigns it a layout that specifies:

• Stripe size: The size of each contiguous chunk written to a single OST, commonly set to 1 MB or 4 MB for scientific workloads.
• Stripe count: The number of OSTs across which the file is distributed.
• Stripe offset: The index of the first OST where striping begins, which is usually chosen automatically to balance load across the cluster.

For a byte at offset \(B\) within a file, Lustre determines the target OST index \(i\) using modular arithmetic:

\[i = \left( \text{offset} + \left\lfloor \frac{B}{S} \right\rfloor \right) \bmod C\]

Modern Lustre deployments also support Progressive File Layouts (PFL). PFL allows a single file to use different striping parameters at different byte ranges. Small files or the initial megabytes of a file might use a stripe count of 1 to minimize overhead, while gigabytes further into the file automatically expand to dozens or hundreds of OSTs.

Striping in GPFS

GPFS, now known as IBM Storage Scale, takes a block-level approach rather than an object-based approach. It organizes storage devices into Network Shared Disks (NSDs). Unlike Lustre, which allows per-file striping configurations, GPFS typically applies striping policies at the file system or storage pool level.

When a file is created in GPFS, its data blocks are distributed across all NSDs in the assigned storage pool using a round-robin or scatter allocation algorithm. The block size in GPFS is often much larger than in traditional file systems, ranging from 256 KB up to 16 MB.

To prevent lock contention when hundreds of compute nodes write to the same file simultaneously, GPFS uses distributed byte-range locking. A token manager grants locks on specific byte ranges rather than the entire file. This allows Node A to write to bytes \(0\) through \(S-1\) on one NSD while Node B writes to bytes \(S\) through \(2S-1\) on another NSD, all in parallel.

Performance Trade-Offs and Alignment

Choosing the right striping parameters is critical for scientific applications using libraries like MPI-IO, HDF5, or NetCDF. If the stripe size \(S\) is too small, the system spends excessive time managing metadata and network connections. If \(S\) is too large, a dataset may not spread across enough targets to utilize the available aggregate bandwidth.

Maximum theoretical throughput \(T_{\text{max}}\) scales with the number of active targets \(C\) and the bandwidth per target \(B_{\text{target}}\), bounded by the client network interface \(B_{\text{network}}\):

\[T_{\text{max}} = \min\left( C \cdot B_{\text{target}}, \, B_{\text{network}} \right)\]

To achieve optimal performance, scientific workflows align their application-level I/O buffers with the underlying file system stripe size. When collective I/O operations match the stripe boundaries, each compute node communicates primarily with a single storage target per I/O phase, minimizing cross-network traffic and maximizing sustained throughput.