Linux Pipe Operator: How Command Chaining Works

The pipe operator is a fundamental feature of the Linux command line that enables users to combine multiple distinct utilities into a unified workflow. This article explains how the pipe operator (|) functions, breaking down its internal mechanisms, how it redirects standard streams between processes, and how you can use it to build efficient command chains.

What is the Linux Pipe Operator?

Represented by the vertical bar symbol (|), the pipe operator redirects the standard output (stdout) of one process directly into the standard input (stdin) of another process. Instead of saving intermediate results to temporary files on disk, the pipe operator transfers data directly through memory, creating a seamless stream from command to command.

The basic syntax is:

command1 | command2

In this setup, command1 produces data, and command2 immediately processes that data as its input.

How the Pipe Works Internally

To understand how pipes work, you need to understand three core concepts in Unix-like operating systems: standard streams, file descriptors, and the pipe() system call.

1. Standard Streams and File Descriptors

Every standard Linux process starts with three default input/output channels, managed as file descriptors:

2. The pipe() System Call

When the shell encounters a pipe character, it requests that the Linux kernel allocate an anonymous, unidirectional data channel in memory using the pipe() system call. The kernel returns two file descriptors: one for reading and one for writing.

3. Process Execution and Concurrency

Unlike scripts that run commands sequentially by waiting for the first command to finish before starting the next, piped processes run concurrently (at the same time):

  1. The shell uses the fork() system call to create child processes for both command1 and command2.
  2. Using the dup2() system call, the shell closes stdout (FD 1) for command1 and points it to the write end of the kernel pipe.
  3. Simultaneously, the shell closes stdin (FD 0) for command2 and points it to the read end of the pipe.
  4. The kernel manages a circular memory buffer (typically 64KB on modern Linux systems) between the two processes. If command1 produces data faster than command2 can consume it, the kernel pauses command1 until space frees up. If command2 needs more data, it waits until command1 writes it.

Because stderr is not redirected by the standard pipe operator, any error messages generated by either command are still printed directly to the terminal screen rather than being fed into the next command.

Practical Examples of Command Chaining

You can chain more than two commands together. The data simply flows through the pipeline from left to right.

Filtering and Searching

To search for running processes without manually scrolling through a long list:

ps aux | grep "nginx"

ps aux outputs all active processes to stdout. The pipe forwards that list directly to grep, which scans the stream and displays only lines containing the term "nginx".

Sorting and Aggregating Data

To find the number of unique IP addresses accessing a web server:

cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr

The data flows through multiple transformations:

  1. cat outputs the log file.
  2. awk extracts only the first column (IP addresses).
  3. sort organizes the IP addresses alphabetically (required for uniq).
  4. uniq -c collapses duplicate lines and prefixes each with a count.
  5. sort -nr sorts the final list numerically in reverse order to show the most frequent visitors at the top.

Paging Long Output

To view system logs one screen at a time:

journalctl -b | less

journalctl generates the boot log, and less receives it as input, giving you an interactive interface to scroll up and down.

Pipe (|) vs. Redirection (>)

It is important to distinguish between piping and redirection:

For example, command1 > output.txt writes data to a file, whereas command1 | command2 sends data directly to command2 via memory. If you want to direct both stdout and stderr through a pipeline, modern shells like Bash support the |& syntax (shorthand for 2>&1 |).