How Safari's Live Text Extracts Text from Images and Videos

This article explores the underlying technology and user mechanics of Apple's Live Text feature within the Safari web browser. It explains how on-device machine learning and optical character recognition work together to detect, isolate, and convert flattened pixels from raster images and paused video frames into interactive text that users can seamlessly select, copy, look up, or translate.

The Core Technology: On-Device OCR

Raster images (such as JPEGs, PNGs, and WebP files) and video frames do not contain actual text data; they are simply grids of colored pixels. To make this visual information interactive, Safari utilizes Apple’s Vision framework, which relies on advanced Optical Character Recognition (OCR) powered by the device's Neural Engine.

Because the processing happens entirely on the device rather than on a remote cloud server, Safari can analyze visual media almost instantaneously while maintaining user privacy.

Step-by-Step Text Detection Process

When you view a webpage or pause a video in Safari, the browser executes a rapid sequence of background tasks to make the text interactive:

  1. Frame Capture: For static images, Safari reads the image file directly. For paused HTML5 videos, Safari captures the exact video frame buffer currently rendered on the screen and treats it as a temporary static raster image.
  2. Region Identification: Deep learning models scan the pixel grid to distinguish background noise from potential characters. The system identifies text blocks regardless of angle, curve, or font style, drawing invisible bounding boxes around words and lines.
  3. Character Recognition: The Neural Engine analyzes the pixel patterns within those bounding boxes, matching them against known alphabets, symbols, and linguistic structures to convert the visual shapes into machine-readable string data.
  4. Overlay Generation: Safari generates an invisible, interactive vector layer directly over the image or paused video frame, mapping the recognized text strings to the exact coordinates of the pixels beneath them.

Interacting with the Extracted Text

Once the invisible overlay is generated, Safari treats the recognized areas just like standard HTML text on a webpage. When a user hovers a mouse cursor over the text on a Mac, the pointer automatically changes into the standard "I-beam" text selection tool. On iOS and iPadOS, tapping and holding the text reveals standard selection handles.

Alternatively, users can click or tap the dedicated Live Text button that appears in the corner of compatible images and paused videos to highlight all detectable text at once.

Executing System Actions

After the pixels are translated into selectable strings via the invisible overlay, Safari integrates with the broader operating system to perform specific actions:

  • Select and Copy: Dragging across the text highlights the mapped coordinates on the overlay. Pressing copy transfers the underlying machine-encoded text string directly to the system clipboard, stripping away the image data so it can be pasted as plain text elsewhere.
  • Look Up: Selecting this option passes the extracted text string to the operating system's internal dictionary and Spotlight search engine. Safari then displays a pop-up card with definitions, Wikipedia entries, and relevant web links.
  • Translate: When triggered, Safari sends the recognized text string to Apple's system-wide Translation framework. The system detects the source language, translates the string into the user's preferred language, and displays the translated text either in a pop-up menu or directly over the original pixels in the image or video frame.

For video content, this entire interactive layer remains active only while the media is paused. The moment the user presses play, Safari discards the temporary overlay and resumes standard video playback.