XML Character Encoding: UTF-8 and UTF-16 Explained

This article explores how the Extensible Markup Language (XML) standard manages character encodings, focusing on UTF-8 and UTF-16. It covers default encoding behaviors, the role of the XML prolog, Byte Order Mark (BOM) auto-detection, and how XML parsers handle character references and encoding mismatches to ensure seamless data exchange across diverse computing environments.

Default Encoding Requirements

The W3C XML specification mandates that all conforming XML processors must natively support both UTF-8 and UTF-16 encodings. If an XML document lacks an explicit encoding declaration and lacks an external protocol specification (such as an HTTP Content-Type header), the parser is required by standard to treat the document as either UTF-8 or UTF-16.

The XML Encoding Declaration

To specify an encoding explicitly, authors use the encoding attribute within the XML declaration (the prolog) at the very beginning of the file:

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <message>Hello, World!</message>
</root>

For UTF-16, the declaration is updated accordingly:

<?xml version="1.0" encoding="UTF-16"?>
<root>
    <message>Hello, World!</message>
</root>

Rules for the encoding declaration include: * It must appear on the first line of the XML document, beginning at the very first byte (unless preceded by a Byte Order Mark). * The encoding name is case-insensitive (e.g., UTF-8, utf-8, and Utf-8 are equivalent). * It must match the actual encoding of the underlying byte stream; declaring UTF-8 for a file saved as ISO-8859-1 causes parsing errors when non-ASCII bytes are encountered.

Auto-Detection and the Byte Order Mark (BOM)

XML processors use an auto-detection algorithm to determine the encoding of the first few bytes before the declaration itself can be read. UTF-16 documents typically begin with a Byte Order Mark (BOM):

• UTF-16 Big-Endian (BE): Begins with byte sequence 0xFE 0xFF
• UTF-16 Little-Endian (LE): Begins with byte sequence 0xFF 0xFE

If a BOM is present, the parser automatically identifies the stream as UTF-16 and detects its endianness.

For UTF-8, a BOM (0xEF 0xBB 0xBF) is permitted but not required by the standard. If no BOM is present, the parser checks the initial ASCII bytes (like <?xml) to confirm an 8-bit encoding family, defaulting to UTF-8.

Handling Characters Outside the Declared Encoding

When an author needs to include a Unicode character that cannot be directly represented in the document’s encoding, XML provides numeric character references (NCRs). These allow any valid Unicode code point to be represented using ASCII-safe sequences:

• Decimal format: © (Copyright sign, ©)
• Hexadecimal format: © or € (Euro sign, €)

Because numeric character references resolve to Unicode scalar values regardless of the file’s binary encoding, they ensure character integrity even if the transport layer alters the text encoding.

Parser Compliance and Error Handling

The XML standard enforces strict well-formedness constraints regarding encoding:

1. Fatal Errors: If an XML parser detects an illegal byte sequence for the declared encoding (such as an invalid multi-byte sequence in UTF-8), it must immediately halt processing and raise a fatal error.
2. Unsupported Encodings: If a document declares an encoding that the parser does not support (such as certain legacy code pages), parsing must fail.
3. No Guessing: Unlike web browsers parsing lenient HTML, XML processors are not allowed to guess the character encoding or attempt silent recovery from encoding errors.