Disambiguating Names with Entity Linking in AI

Entity linking is the artificial intelligence process of connecting ambiguous text mentions to their exact, corresponding entries in a structured knowledge graph like Wikidata. When a text mentions "Paris" or "Michael Jordan," an AI system must determine whether the text refers to the French capital or a celebrity, or the basketball legend versus the machine learning professor. This article explains the exact pipeline AI uses to disambiguate surface names against millions of Wikidata nodes: mention detection, candidate generation, contextual disambiguation, and graph coherence.

1. Mention Detection (Named Entity Recognition)

Before an AI can link a name to Wikidata, it must find the name in the raw text. A Named Entity Recognition (NER) model scans the input sentence and identifies spans of text that represent real-world objects, people, or places.

For example, in the sentence, "Michael Jordan published a new paper on neural networks," the NER system highlights "Michael Jordan" as a surface mention requiring linking.

2. Candidate Generation

Wikidata contains over 100 million items, each assigned a unique Q-identifier (QID). Comparing a single text mention against 100 million nodes simultaneously is computationally impossible. Candidate generation narrows the search space from millions of nodes down to a few dozen plausible options.

The system uses three fast retrieval methods to build this list:

  • Alias Dictionaries: Wikidata stores labels and "also known as" aliases for every node. The system searches this index for exact or fuzzy matches.
  • Prior Probability: The AI looks at historical data (such as Wikipedia hyperlink frequencies) to see which Wikidata nodes are most commonly associated with a specific text string.
  • Dense Retrieval: The text mention is converted into a mathematical vector (embedding) and matched against pre-computed vectors of Wikidata entities.

For "Paris," candidate generation pulls Q90 (capital of France), Q165116 (Paris Hilton), and Q167646 (mythological Trojan prince). For "Michael Jordan," it pulls Q41421 (the NBA player) and Q3308285 (the Berkeley professor).

3. Contextual Disambiguation

Once the AI has a shortlist of 10 to 50 candidate QIDs, it uses deep learning—specifically Transformer models like BERT—to evaluate the context surrounding the mention. This is commonly done using a Bi-Encoder followed by a Cross-Encoder.

Bi-Encoder (Fast Filtering)

Two separate neural networks run in parallel. One encodes the input sentence containing the mention into a vector. The other encodes the Wikidata description and properties of the candidate nodes into vectors. The system calculates the dot product (similarity) between the sentence vector and the candidate vectors.

Cross-Encoder (Deep Ranking)

To make the final decision, the Cross-Encoder concatenates the input sentence and the Wikidata candidate description into a single string and processes them together. This allows the AI's self-attention mechanism to directly compare words in the text against attributes in the knowledge graph.

Look at how context shifts the mathematical weights:

  • Input: "Michael Jordan published a new paper on neural networks."
  • Candidate A (Q41421): Description is "American basketball player." The words "basketball" and "player" have low semantic similarity with "paper" and "neural networks."
  • Candidate B (Q3308285): Description is "American scientist, professor in machine learning." The words "scientist" and "machine learning" have high semantic attention scores with "paper" and "neural networks."

The Cross-Encoder assigns the highest probability score to Q3308285.

4. Global Graph Coherence

Text rarely contains just one entity. AI systems use the surrounding entities in a paragraph to enforce "global coherence" across the knowledge graph. The AI checks how closely connected the candidate nodes are within Wikidata's structural web of properties (like instance of, located in, or part of).

Consider the sentence: "Paris went to the Louvre."

  • "Paris" yields Q90 (the city) and Q165116 (Paris Hilton).
  • "Louvre" yields Q19675 (the art museum in Paris, France).

In Wikidata, Q19675 (Louvre) has a direct relational edge to Q90 (Paris, France) via the location property. However, the grammatical structure ("went to") indicates an animate actor. By combining the linguistic context from the Transformer with the structural edges of the knowledge graph (often using Graph Neural Networks), the AI determines whether the entities align geographically or socially.

Conversely, in "I took a train from Lyon to Paris," the node for "Lyon" (Q456) shares country, railway, and geographic properties with "Paris" (Q90). The graph distance between Q456 and Q90 is extremely short, locking in Q90 as the correct canonical node.

5. Handling "NIL" Entities

Sometimes, a surface mention refers to a person or place that does not exist in Wikidata yet. During the disambiguation phase, if the highest-scoring candidate from the Cross-Encoder falls below a strict confidence threshold, the system refuses to link it to an existing node. Instead, it flags the mention as a NIL (Not In Lexicon) entity, preventing false merges and signaling that a new node may need to be created in the knowledge graph.