How to Use Linux Parallel to Run Jobs Concurrently

The GNU parallel command in Linux is a command-line utility designed to execute tasks simultaneously across multi-core processors, reducing the overall execution time of batch operations. By splitting standard input or argument lists across multiple CPU cores, parallel serves as a high-performance alternative to traditional shell loops and tools like xargs. This article explains how the Linux operating system uses GNU parallel, its underlying execution mechanisms, and the practical syntax needed to run jobs concurrently.

Understanding GNU Parallel

GNU parallel operates by reading items from standard input (stdin) or argument lists, wrapping each item into a separate shell command, and launching these commands as independent processes. Instead of processing items sequentially one after another, it automatically discovers the number of available CPU cores and spawns one worker process per CPU thread by default. This ensures the operating system utilizes available hardware resources without overburdening the CPU scheduler.

Core Execution Mechanics

When a user invokes parallel, the utility manages concurrency through three primary phases:

1. Input Splitting and Tokenization: The command divides the input data into individual arguments based on delimiters (typically newlines). It places these arguments into placeholders, represented by {} by default.
2. Process Orchestration: The tool monitors the active process table. As soon as a process finishes execution, GNU parallel immediately launches the next queued task to keep all configured CPU threads saturated.
3. Output Aggregation: Running multiple commands simultaneously usually results in scrambled terminal output. GNU parallel buffers stdout and stderr for each job, printing the complete output only when an individual job terminates. This guarantees deterministic, uncorrupted logs.

Basic Syntax and Usage

The most direct way to pass arguments to parallel is using the ::: separator:

parallel gzip ::: file1.txt file2.txt file3.txt

In this command, gzip runs against file1.txt, file2.txt, and file3.txt simultaneously, provided there are at least three available CPU threads.

You can also pipe input directly from files or command outputs:

cat urls.txt | parallel curl -O {}

Here, {} is automatically replaced by each line from urls.txt.

Controlling Concurrency Levels

While parallel defaults to running one job per CPU core, you can manually control how many tasks execute at the same time using the -j or --jobs flag:

• Fixed number of jobs: Specifying -j 4 forces the system to run exactly four jobs simultaneously, regardless of core count.
• Proportional to CPU cores: Setting -j +2 instructs the system to run two more jobs than the number of detected CPU cores. Setting -j 50% utilizes half of the available cores.
• Infinite concurrency: Specifying -j 0 runs all jobs at once without limiting concurrency, which should be used with caution to prevent resource exhaustion.

Managing Arguments and Output Ordering

When operations depend on strict sequencing or require complex parameters, GNU parallel provides built-in modifiers:

• --keep-order (-k): Forces parallel to output results in the exact order of the input list, even if a later job finishes ahead of an earlier one.
• Argument Modifiers: Placeholders can strip extensions ({.}) or extract directory names ({//}). For example:

  parallel convert {} {.}.png ::: *.jpg

• Multiple Inputs: Using multiple ::: separators creates a Cartesian product of the inputs:

  parallel echo {1} and {2} ::: A B ::: 1 2

  This runs four jobs: A and 1, A and 2, B and 1, and B and 2.

Why Linux Uses Parallel Over Standard Shell Loops

Standard for or while loops in Bash execute commands sequentially in a single thread. While appending an ampersand (&) to a command pushes it into the background, doing so inside a large loop creates thousands of simultaneous processes, leading to CPU thrashing, memory exhaustion, and potential system crashes. GNU parallel acts as a managed queue, continuously maintaining a stable workload that maximizes CPU throughput while keeping system memory and processor load within predictable limits.