Curl URL Globbing for Downloading File Sequences

URL globbing in curl is a powerful feature that allows users to download multiple files matching a specific sequence or pattern using a single command. By utilizing brackets and braces, curl can automatically iterate through numerical ranges, alphabetical sequences, or defined lists of words. This article explains how curl handles these patterns, how to output the files correctly, and how to disable the feature when necessary.

Understanding Curl Globbing Syntax

Curl supports two primary types of globbing characters: square brackets [] for ranges and curly braces {} for lists.

1. Numerical Ranges

To download a sequence of files numbered in order, use square brackets containing the start and end values:

curl http://example.com/file[1-10].jpg

You can also specify a step counter to skip numbers. For example, to download every second file in a range:

curl http://example.com/file[1-10:2].jpg

This command matches file1.jpg, file3.jpg, file5.jpg, file7.jpg, and file9.jpg. Zero-padding is also supported; using [01-10] will preserve the leading zeros in the requests.

2. Alphabetical Ranges

Alphabetical sequences work similarly to numerical ranges. Curl can iterate through lowercase or uppercase letters:

curl http://example.com/archive-[a-z].zip

3. Lists of Words

If the files do not follow a strict sequential range but share a pattern, you can use curly braces to specify a comma-separated list of specific values:

curl http://example.com/{image,document,video}.tar

Saving Sequential Downloads

When downloading multiple files, you must instruct curl how to save them so they do not overwrite each other or stream directly to the terminal stdout.

Using Match Variables

The lowercase -o option allows you to define a custom output filename. Curl uses #1, #2, #3, etc., as placeholders to represent the match value of the corresponding globbed pattern.

curl http://example.com/photo-[1-5].jpg -o "local_photo_#1.jpg"

In this example, #1 is replaced by the number from the first globbing pattern. If you use multiple patterns in a single URL, #2 refers to the second pattern:

curl http://example.com/[2021-2023]/report-{jan,feb}.pdf -o "report_#1_#2.pdf"

Using the Remote Name Option

Alternatively, you can use the uppercase -O (or --remote-name) option. When combined with globbing, curl automatically saves each file locally using its original remote filename:

curl -O http://example.com/file[1-10].zip

Disabling Globbing

If your URL contains literal square brackets or curly braces (such as in complex API query parameters) that should not be interpreted as globbing patterns, you must disable the feature using the -g or --globoff flag:

curl -g "http://example.com/search?q=[term]"