Using uniq -c to Count Duplicate Lines in Linux

This article provides a comprehensive overview of the uniq -c command in the Linux operating system, detailing its primary purpose, how it operates, and why it is a critical tool for text processing. You will learn the mechanics behind prefixing duplicate lines with occurrence counts, the essential requirement of sorting input data prior to using the command, and practical examples of combining uniq -c with other command-line utilities for data and log analysis.

The Purpose of uniq -c

The primary purpose of the uniq command in Linux is to filter out repeated adjacent lines from a text file or standard input stream. When the -c (or --count) flag is added, the command changes its default behavior: instead of merely suppressing duplicate entries, it prefixes each unique output line with the number of times that specific line occurred consecutively in the input.

This makes uniq -c an essential utility for frequency analysis, auditing log files, identifying common patterns, and generating summary reports directly from the terminal.

The Adjacency Requirement and the Role of sort

A fundamental characteristic of the uniq command is that it only evaluates adjacent lines. It reads input sequentially and compares each line strictly to the one immediately preceding it. If identical lines are scattered throughout different parts of a file, uniq -c will treat each separated occurrence as a new, distinct entry rather than aggregating them into a single count.

To count all duplicate lines across an entire file, the data must first be organized using the sort command. Piping the output of sort into uniq -c groups all identical lines together, allowing uniq -c to calculate an accurate total frequency for every line:

sort filename.txt | uniq -c

Basic Syntax and Output Format

The standard syntax for using uniq -c is:

uniq -c [input_file [output_file]]

When executed, the output displays in two columns: the count of occurrences padded with spaces on the left, followed by the actual text content of the line on the right.

For example, consider a file named fruits.txt with the following content:

apple
banana
apple
orange
banana
apple

Running sort fruits.txt | uniq -c produces:

      3 apple
      2 banana
      1 orange

Common Use Cases and Advanced Combinations

The uniq -c command is most powerful when chained with other core Linux text-processing tools:

• Sorting by Frequency: After counting duplicates, you can pipe the output into sort -n to sort by count in ascending order, or sort -nr to sort in descending order. This is commonly used to find the most frequent occurrences (such as the top IP addresses in a web server access log):

  cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10

• Case-Insensitive Counting: Combining the -i option with -c (uniq -ci) allows you to count occurrences while ignoring differences in character case (treating "Apple" and "apple" as the same entry).
• Displaying Only Duplicates: If you only want to view lines that appear more than once along with their counts, pair -c with the -d option:

  sort filename.txt | uniq -cd