Linux Software-Defined Storage with Ceph

This article provides an overview of how the Linux operating system implements software-defined storage (SDS) through Ceph, transforming commodity hardware into an enterprise-grade, distributed storage cluster. It explores the core architecture of Ceph on Linux, the underlying Reliable Autonomic Distributed Object Store (RADOS), algorithmic data distribution using CRUSH, and how the Linux kernel and user-space subsystems expose object, block, and file storage interfaces.

The Foundation: RADOS and Linux Daemons

At the core of Ceph's implementation on Linux is the Reliable Autonomic Distributed Object Store (RADOS). RADOS runs entirely in user-space as a collection of specialized Linux system daemons:

• Ceph OSD (Object Storage Daemon): Manages local storage drives, handles data replication, recovery, and rebalancing. Modern deployments run OSDs using BlueStore, an internal storage engine that bypasses the traditional Linux VFS (Virtual File System) to write directly to raw block devices using asynchronous I/O (libaio), eliminating double-journaling penalties.
• Ceph Monitor (MON): Maintains cluster state maps using a Paxos consensus algorithm to ensure consistency and quorum across nodes.
• Ceph Manager (MGR): Provides monitoring, metrics collection, and management interfaces, tracking cluster performance and hardware health.

These daemons integrate natively with the Linux operating system via systemd unit files, standard networking stacks (TCP/IP and RDMA), and control groups (cgroups) for resource isolation.

Algorithmic Data Distribution: The CRUSH Engine

Unlike traditional storage architectures that rely on centralized metadata lookup tables, Linux systems running Ceph use the CRUSH (Controlled Replication Under Scalable Hashing) algorithm.

CRUSH computes the exact physical location of data deterministically based on an object's identifier and the cluster’s topology map. Because clients and storage nodes independently compute where data resides using local CPU cycles, the Linux network stack is freed from routing constant lookup requests to a centralized metadata server. This architecture eliminates bottlenecks and allows the cluster to scale horizontally across thousands of Linux nodes.

Linux Kernel and User-Space Integration

Ceph integrates into the Linux ecosystem at multiple abstraction layers depending on the required storage type:

1. Block Storage (RBD)

Ceph provides the RADOS Block Device (RBD) via two distinct pathways in Linux:

• Kernel Client (rbd.ko): A native Linux kernel module exposes distributed storage as a standard block device (e.g., /dev/rbd0). This device can be formatted with standard Linux filesystems like Ext4 or XFS and mounted like any local disk.
• User-space Driver (librbd): Provides direct integration with virtualization stacks such as QEMU/KVM. By bypassing the host Linux kernel's block layer, virtual machines running on Linux can read and write directly to the Ceph cluster over network sockets, reducing CPU context switching and latency.

2. File Storage (CephFS)

CephFS delivers a POSIX-compliant distributed file system running on top of RADOS, assisted by Ceph Metadata Server (MDS) daemons:

• Kernel Driver (ceph.ko): Integrated into the mainline Linux kernel, this driver allows native Linux systems to mount CephFS using standard mount -t ceph commands, tying directly into the Linux Virtual File System (VFS).
• User-space Client (ceph-fuse): Mounts the filesystem in user space using the Linux FUSE framework, simplifying deployment across various Linux distributions without requiring root-level kernel modules.

3. Object Storage (RADOS Gateway)

The RADOS Gateway (RGW) is a user-space daemon that exposes RESTful APIs compatible with Amazon S3 and OpenStack Swift. It translates standard HTTP/HTTPS requests processed through the Linux socket layer directly into native RADOS object operations.

High Availability and Self-Healing

The Linux implementation of Ceph relies on active peer-to-peer communication among OSDs. Storage nodes continuously monitor one another via network heartbeats. If a Linux host or drive fails, the surviving OSDs detect the loss, update the cluster map, and initiate background data reconstruction automatically to restore the desired replication or erasure-coding state without administrator intervention.