How AI Links Pronouns to Entities Across Paragraphs

This article provides a comprehensive overview of how coreference resolution in natural language processing (NLP) enables artificial intelligence to connect pronouns and noun phrases to the same real-world entity across multiple paragraphs. We will explore the mechanics of mention detection, the role of vector embeddings, how transformer models process long-range context, and the clustering algorithms used to group these references together accurately.

Coreference resolution is the task of finding all expressions that refer to the same entity in a text. For an AI to understand a long document, it must know that "Dr. Jane Smith" in paragraph one, "the lead researcher" in paragraph two, and "she" in paragraph three all point to the same person. Achieving this across paragraphs requires a multi-step pipeline driven by deep learning.

1. Mention Detection

The first step is identifying all potential references to entities, known as "mentions." The AI scans the text to extract named entities (like "Jane Smith"), noun phrases (like "the talented scientist"), and pronouns (like "she," "her," "they"). Modern NLP systems use neural networks to score every possible span of words in the document based on the likelihood that the span represents a valid entity mention.

2. Generating Contextual Embeddings

Once mentions are identified, the AI converts these words into numerical representations called vector embeddings. Modern NLP relies heavily on transformer-based language models (such as BERT, RoBERTa, or GPT) to create these embeddings.

Unlike older models that looked at words in isolation, transformers use a mechanism called "self-attention." Self-attention allows the AI to look at a pronoun in the fourth paragraph and simultaneously weigh its relationship to every other word in the preceding paragraphs. The resulting vector for the word "she" does not just represent a female pronoun; it mathematically encodes the context surrounding it, making it geometrically similar to the vector for "Dr. Jane Smith" generated earlier in the text.

3. Handling Long-Range Context

A major challenge in cross-paragraph resolution is that standard transformer models have a limit on how much text they can process at once. To link entities across many paragraphs, AI systems use two main strategies:

  • Overlapping Windows: The AI breaks the document into smaller, overlapping chunks of paragraphs. It processes the first chunk, keeps the most important entity data in a memory buffer, and passes that memory into the next chunk.
  • Long-Context Transformers: Newer models (like Longformer or BigBird) use sparse attention mechanisms. Instead of comparing every word to every other word—which requires massive computing power—they selectively focus on local words and a few globally important words (like main subjects), allowing them to process thousands of words across dozens of paragraphs in a single pass.

4. Scoring Compatibility

With contextual embeddings established across the paragraphs, the AI evaluates pairs of mentions to see if they match. It passes the vectors of an antecedent (the earlier mention, e.g., "Apple Inc.") and an anaphor (the later mention, e.g., "the tech giant" or "it") through a neural network to calculate a compatibility score.

During this phase, the model implicitly checks for several linguistic factors learned during its training:

  • Number and Gender Agreement: Matching "she" to a female entity, or "they" to a plural noun.
  • Semantic Compatibility: Knowing that "the company" can refer to "Microsoft," but not to "Bill Gates."
  • Distance and Salience: Weighing how far apart the mentions are, while recognizing that a main topic introduced in the first paragraph remains a likely match for pronouns much later in the text.

5. Entity Clustering

Finally, the AI uses the compatibility scores to group mentions into clusters. If "Dr. Smith" (Paragraph 1) links strongly to "the researcher" (Paragraph 2), and "the researcher" links strongly to "she" (Paragraph 3), the algorithm merges all three into a single entity cluster. Every item in this cluster is then recognized by the AI as referring to the exact same real-world entity, allowing the system to maintain a coherent understanding of the entire document.