Linux Variable Expansion and Parameter Substitution

Variable expansion and parameter substitution are fundamental mechanisms in Linux shell environments that allow scripts to dynamically evaluate, manipulate, and inject values before executing commands. When a shell like Bash or Dash encounters a dollar sign ($), it initiates an expansion phase that replaces variable names with their stored values or applies inline transformation logic. This article breaks down how the Linux shell parses these expressions, handles conditional fallback values, executes pattern-based string manipulations, and fits substitution into its broader command execution lifecycle.

The Mechanism of Variable Expansion

In Linux shell scripting, a parameter is an entity that stores values, which can be a name (variable), a number (positional parameter like $1), or a special symbol (such as $? or $$). Variable expansion is the process of retrieving the value assigned to that parameter.

The most basic form uses the $ prefix:

NAME="Linux"
echo $NAME

When unambiguous separation is required between the variable name and adjacent characters, curly braces are mandatory:

PREFIX="super"
echo "${PREFIX}user" # Outputs: superuser

Without the braces, the shell attempts to evaluate $PREFIXuser, which likely returns an empty string if undefined.

Parameter Substitution Operations

Parameter substitution extends basic expansion by allowing conditional checks, default value assignments, and string manipulation directly within the ${} syntax, eliminating the need for external tools like awk or sed.

1. Conditional and Fallback Expansions

Shells provide conditional operators to handle unset or null variables:

• Default Value (${VAR:-default}): Uses default if VAR is unset or null. The original variable remains unchanged.
• Assign Default (${VAR:=default}): If VAR is unset or null, it sets VAR to default and expands to that value.
• Display Error (${VAR:?error_message}): Throws an error with error_message and terminates the script if VAR is unset or null.
• Alternate Value (${VAR:+alternate}): Expands to alternate only if VAR is set and not null; expands to nothing otherwise.

2. String Length and Slicing

The shell can compute string length or extract substrings natively:

• String Length (${#VAR}): Returns the number of characters in the value of VAR.
• Substring Extraction (${VAR:offset:length}): Extracts a slice starting at the zero-based index offset for a specified length. If length is omitted, it extracts to the end of the string.

TEXT="OperatingSystem"
echo "${TEXT:0:9}" # Outputs: Operating

3. Pattern Removal (Trimming)

Shell parameter substitution allows prefix and suffix truncation using standard glob patterns:

• Remove Shortest Prefix (${VAR#pattern}): Deletes the shortest match of pattern from the beginning of VAR.
• Remove Longest Prefix (${VAR##pattern}): Deletes the longest match of pattern from the beginning of VAR. Commonly used to get file basenames.
• Remove Shortest Suffix (${VAR%pattern}): Deletes the shortest match of pattern from the end of VAR. Commonly used to strip file extensions.
• Remove Longest Suffix (${VAR%%pattern}): Deletes the longest match of pattern from the end of VAR.

FILE="/path/to/archive.tar.gz"
echo "${FILE##*/}" # Outputs: archive.tar.gz
echo "${FILE%.*}"  # Outputs: /path/to/archive.tar

4. Pattern Replacement

Bash and compatible shells support search-and-replace transformations:

• Replace First Match (${VAR/pattern/replacement}): Replaces the first match of pattern with replacement.
• Replace All Matches (${VAR//pattern/replacement}): Replaces every match of pattern with replacement.
• Anchor Replacement (${VAR/#pattern/replacement} or ${VAR/%pattern/replacement}): Restricts the substitution to the prefix (#) or suffix (%) of the string.

Order of Shell Expansion

The Linux shell does not evaluate expressions at random. It follows a strict, sequential order of operations before passing arguments to a command:

1. Brace Expansion: Expressions like a{b,c} expand to ab ac.
2. Tilde Expansion: ~ expands to the user's home directory.
3. Parameter and Variable Expansion: ${VAR} substitutions occur.
4. Command Substitution: Expressions like $(command) or `command` are executed, and their output replaces the expression.
5. Arithmetic Expansion: Expressions inside $(( ... )) are computed.
6. Word Splitting: Unquoted expansions are scanned according to the Internal Field Separator (IFS, usually space, tab, newline) and split into distinct arguments.
7. Pathname Expansion (Globbing): Patterns like * or ? are replaced with matching filenames.
8. Quote Removal: Single and double quotation marks used to suppress previous expansions are removed.

Because word splitting occurs immediately after parameter expansion, failing to wrap variables in double quotes ("$VAR") can cause variables containing whitespace to split into multiple distinct arguments, often resulting in scripting bugs or security vulnerabilities. Double-quoting preserves the integrity of the expanded text as a single word.