How itertools.groupby() Groups Items in Python

Python's itertools.groupby() creates consecutive chunks of elements from an iterable based on a shared key or identity. This article explains the core mechanism behind groupby(), why pre-sorting your dataset is critical, how to utilize the key parameter, and how to avoid the common pitfalls associated with lazy evaluation.

The Core Mechanism: Consecutive Grouping

Unlike the GROUP BY clause in SQL, Python's itertools.groupby() does not look across the entire iterable to gather matching elements into single buckets. Instead, it processes items sequentially and splits the stream whenever the calculated key changes.

When itertools.groupby() iterates through a sequence, it evaluates each element using an optional key function. If the current item has the same key as the previous item, it appends the item to the current group. As soon as an element with a different key is encountered, the current group is closed, and a new group is started.

The Requirement to Sort

Because grouping only occurs over consecutive matches, non-adjacent identical keys will produce separate, duplicate groups. To group all identical elements together across an entire collection, you must first sort the iterable using the same key function.

Consider this unsorted list:

import itertools

data = ["apple", "apricot", "banana", "avocado"]

for key, group in itertools.groupby(data, key=lambda x: x[0]):
    print(key, list(group))

Output:

a ['apple', 'apricot']
b ['banana']
a ['avocado']

Because "banana" interrupted the sequence, the letter 'a' appears twice as a group key. To ensure all items starting with 'a' reside in a single group, sort the list first:

import itertools

data = ["apple", "apricot", "banana", "avocado"]
sorted_data = sorted(data, key=lambda x: x[0])

for key, group in itertools.groupby(sorted_data, key=lambda x: x[0]):
    print(key, list(group))

Output:

a ['apple', 'apricot', 'avocado']
b ['banana']

Syntax and Key Functions

The syntax for the function is:

itertools.groupby(iterable, key=None)

Grouping Complex Objects

The key parameter is frequently used with dictionary items, tuples, or object attributes:

from itertools import groupby
from operator import itemgetter

users = [
    {"name": "Alice", "role": "Admin"},
    {"name": "Bob", "role": "User"},
    {"name": "Charlie", "role": "Admin"},
    {"name": "David", "role": "User"}
]

# Sort by the role first
sorted_users = sorted(users, key=itemgetter("role"))

# Group by role
grouped_users = {}
for role, group in groupby(sorted_users, key=itemgetter("role")):
    grouped_users[role] = list(group)

print(grouped_users)

Lazy Iteration and Pitfalls

itertools.groupby() operates lazily to remain memory-efficient with large datasets. It yields a 2-tuple on each iteration: (key, group_iterator).

The group_iterator shares the underlying iterator of the groupby object. Consequently, when the main groupby loop advances to the next group, the previous group_iterator is exhausted.

# Problematic: Trying to read groups after the loop advances
groups = [group for key, group in itertools.groupby(sorted_data, key=lambda x: x[0])]

# All sub-iterators except the last one will be empty
print([list(g) for g in groups])
# Output: [[], ['banana']]

To retain group members for later use, convert each group iterator into a concrete collection, such as a list, during each iteration of the loop:

# Correct approach
saved_groups = {key: list(group) for key, group in itertools.groupby(sorted_data, key=lambda x: x[0])}