How Safari's Copy Subject Feature Isolates Image Objects

Safari’s "Copy Subject" feature allows users to instantly extract the main object of an image away from its background with a single right-click or long-press. This article provides an overview of how this tool works under the hood, detailing the combination of Apple's Vision framework, on-device machine learning algorithms, semantic segmentation, and hardware acceleration via the Neural Engine that processes and cuts out the image in milliseconds.

Triggering the Vision Framework

When you right-click on Mac or long-press an image on iOS and iPadOS within Safari, the browser does not send the image to a cloud server for processing. Instead, Safari passes the image data directly to the operating system's built-in VisionKit and Core ML frameworks. Before the context menu even finishes animating on your screen, the system quickly analyzes the image to determine if a distinct foreground subject actually exists. If it detects one, the "Copy Subject" option is made available.

Step 1: Saliency Detection

The first phase of isolating the object is called saliency detection. The machine learning model scans the image to determine what naturally draws the human eye. It looks for sharp focus, contrast differences, spatial positioning, and recognizable patterns like human faces, animals, or prominent objects. This creates a rough "heat map" of the image, telling the system where the primary subject is located and what part of the image is merely background noise.

Step 2: Semantic Segmentation

Once the general area of the subject is located, the system uses a computer vision technique called semantic segmentation. Apple’s machine learning models have been trained on millions of labeled images to understand what different objects look like. During this step, the algorithm evaluates the image at the pixel level, classifying every single pixel as either part of the "foreground subject" or part of the "background." This creates a distinct boundary line around the object.

A simple pixel classification often leaves jagged edges, especially around complex details like human hair, animal fur, or semi-transparent objects. To solve this, the algorithm applies an advanced refinement process known as alpha matting.

The model generates a high-resolution alpha mask—a grayscale map where pure white represents the solid subject, pure black represents the background to be deleted, and shades of gray represent partial transparency. This allows Safari to accurately capture fine strands of hair and soft edges without including chunks of the background color.

Hardware Acceleration via the Neural Engine

Performing saliency detection, pixel-by-pixel segmentation, and alpha matting requires billions of mathematical operations. Safari achieves this instantaneously by offloading the task from the main CPU to the Apple Neural Engine (found in Apple Silicon M-series and A-series chips). The Neural Engine is purpose-built to run these specific Core ML matrix multiplications concurrently, reducing a process that used to take minutes in professional photo editing software down to a fraction of a second.

Delivering the Final Cutout

Once the alpha mask is finalized by the Neural Engine, the system strips away all pixels designated as the background, replacing them with transparency. Safari then packages the remaining foreground pixels into a Portable Network Graphics (PNG) file—a format that natively supports transparent backgrounds—and places it directly onto your device's universal clipboard, ready to be pasted into another application.