Why XML Parses Entities Before Markup Tokenization

This article explains why the XML specification mandates that entity and character reference parsing occur before final markup tokenization. It details how this processing order enables structural modularity, distinguishes literal data from syntactic delimiters, and ensures deterministic parsing across XML processors.

Modularity and Dynamic Content Insertion

In XML, general parsed entities are intended to act as structural macros. They allow authors to break large documents into reusable modular components or external files that contain elements, attributes, and text.

If the parser tokenized markup before resolving entities, the replacement text inserted by an entity would be treated as raw character data rather than active markup. By resolving entities first, the processor can inject the replacement text directly into the token stream, allowing child elements, nested tags, and attributes within the entity to be correctly recognized as structural nodes in the document tree.

Disambiguating Syntax from Data

A primary requirement of XML is strict disambiguation between syntactic delimiters (such as < and &) and literal character values. The two-phase relationship between entity replacement and tokenization provides this boundary:

1. Character References as Literal Data: Numeric character references (such as < for <) are processed to represent character data rather than structural delimiters. The specification ensures that expanding a character reference does not inadvertently generate new markup tokens in the same parsing context, preserving literal data integrity.
2. Entity References as Structural Markup: In contrast, parsed general entities are explicitly designed to introduce structural markup.

Establishing clear evaluation rules prior to tokenization ensures the parser knows whether a given character acts as an operational delimiter or plain payload.

Deterministic Validation and Parsing

Validating XML processors must construct a complete, unambiguous representation of the document to evaluate it against Document Type Definitions (DTDs) or schemas. Performing entity expansion prior to the structural evaluation of the grammar guarantees that:

• Well-formedness constraints are applied to the fully realized document hierarchy.
• Contextual constraints (such as an entity opening a tag that it does not close) can be detected and flagged as errors.
• The parser avoids unpredictable state transitions that would occur if tokenization and entity replacement operated concurrently without strict precedence.