How Safari Visual Look Up Identifies Objects in Photos

This article explains the technological process behind Apple's Visual Look Up feature in the Safari web browser. It covers how on-device machine learning, the Apple Neural Engine, and cloud-based databases work together to accurately identify landmarks, plants, dog breeds, and artwork within webpage photos while protecting user privacy.

On-Device Image Detection When you view a webpage in Safari, your device does not automatically send the page's images to the cloud. Instead, the process begins locally when you interact with an image by long-pressing it on iOS or right-clicking it on macOS. The device’s built-in Neural Engine immediately analyzes the photo on the device. Deep learning models built directly into the operating system scan the image's pixels to detect whether a recognizable subject—such as a dog, a flower, a painting, or a famous building—is present, and separates that subject from the background.

Creating a Mathematical Fingerprint Once the on-device system detects a relevant object, it isolates that specific region of the photo. A specialized neural network then analyzes the visual characteristics of the subject, such as the petal arrangement of a plant, the facial structure of a dog breed, the architectural lines of a landmark, or the composition of an artwork. The system converts these visual traits into a compact mathematical representation known as a vector embedding. This numerical data acts as a unique digital fingerprint of the object.

Matching with Server Databases To identify the exact species, breed, landmark, or painting, the device must reference a massive catalog of information that is too large to store on a phone or computer. Safari sends the encrypted mathematical fingerprint to Apple’s servers. The servers then compare those numbers against the Siri Knowledge database, which contains millions of indexed profiles for global landmarks, botanical species, animal breeds, and historical art pieces.

Delivering the Results to Safari When the server finds a high-probability match between the image's numerical fingerprint and an entry in its database, it packages the corresponding data and sends it back to your device. Safari then displays this information in a pop-up window directly over the webpage, providing you with the name of the subject, relevant Wikipedia entries, and similar web images.

Privacy Preservation Throughout this entire identification process, the actual photo from the Safari webpage never leaves your device. Because the communication between your browser and Apple's servers relies strictly on encrypted numerical vectors rather than raw image files, Apple cannot see the photo you are analyzing, nor does it know which website you are visiting.