How AI Instance Segmentation Separates Overlapping Objects

Instance segmentation in artificial intelligence distinguishes between separate, overlapping objects of the same class by combining bounding box object detection with pixel-level semantic segmentation. This article explores the core mechanisms behind this process, including two-stage localization using Region Proposal Networks, precise feature alignment via RoIAlign, and bottom-up pixel clustering techniques that assign unique identifiers to individual entities even when they partially occlude one another.

To understand how instance segmentation handles overlap, it must be differentiated from semantic segmentation. Semantic segmentation classifies every pixel in an image into a category, meaning two overlapping cars are merged into a single blob of "car" pixels. Instance segmentation resolves this by assigning both a class label and a unique instance ID to every pixel. It achieves this separation primarily through two distinct methodologies: top-down (detect-then-segment) and bottom-up (label-then-cluster) approaches.

The most common method for separating overlapping objects is the top-down approach, exemplified by architectures like Mask R-CNN. Before the AI attempts to color in the pixels of an object, it first locates the individual objects using a Region Proposal Network (RPN). The RPN scans the image and generates multiple bounding boxes (Regions of Interest, or RoIs) around potential objects. Even if two people are standing directly in front of one another, the network relies on distinct structural cues—such as two separate heads or pairs of shoulders—to generate two distinct, overlapping bounding boxes.

Once the AI has established separate bounding boxes for the overlapping objects, it processes each box independently. A critical component called RoIAlign extracts the visual features from within a specific bounding box and warps them into a fixed size without losing spatial accuracy. Because the features are isolated to that specific box, the subsequent segmentation branch only has to make a binary decision for each pixel inside that box: is this pixel part of the foreground object, or is it the background? In the context of Box A, the overlapping object from Box B is treated merely as background noise and is ignored.

Toprevent the AI from accidentally merging two overlapping objects that share nearly identical bounding boxes, algorithms use Non-Maximum Suppression (NMS) alongside spatial attention mechanisms. Standard NMS deletes redundant boxes, while advanced variations like Soft-NMS decay the confidence scores of overlapping boxes rather than deleting them outright. This allows the network to maintain separate tracking for closely packed objects of the same class, such as a flock of birds or a pile of fruit.

The alternative method for resolving overlap is the bottom-up approach, which does not use bounding boxes. Instead, the AI performs semantic segmentation first and then groups the pixels into instances using metric learning and pixel embeddings. The network maps every pixel into a high-dimensional mathematical space. It is trained to push pixels belonging to the same object close together in this space, while pushing pixels of different objects—even of the same class—far apart.

Along with embeddings, bottom-up models often predict directional vectors for each pixel. Each pixel essentially points toward the geometric center (centroid) of the specific object it belongs to. When two objects overlap, the pixels along the occlusion boundary will point in completely different directions—one set pointing to the center of the foreground object, and the adjacent set pointing to the center of the background object. A clustering algorithm then groups the pixels based on these directional cues and embedding distances, cleanly slicing the overlapping objects into separate instances.

Finally, modern transformer-based architectures use object queries to handle overlap. Rather than scanning the image with thousands of boxes, the model uses a fixed set of learnable queries, where each query is trained to hunt for a single object instance. Through self-attention and cross-attention mechanisms, these queries communicate with one another to ensure two queries do not lock onto the same object. Each query then directly predicts a pixel mask for its assigned object, naturally separating overlapping instances by leveraging the global context of the entire image.