How UEFI Handles Unicode String Manipulation

This article explores how the Unified Extensible Firmware Interface (UEFI) manages Unicode strings internally across its architecture and execution environments. It covers the underlying encoding standard, memory representation, the role of core protocols such as the Unicode Collation Protocol, case folding, and the internal handling of character comparisons and conversions within EDK II implementations.

Character Encoding: The UCS-2 Standard

UEFI relies primarily on UCS-2 (Universal Character Set 2) encoding rather than standard UTF-8 or full UTF-16, a design choice originating from the legacy EFI 1.10 specifications.

  • Data Type: Strings are composed of elements of the CHAR16 data type, defined as an unsigned 16-bit integer (uint16_t).
  • Fixed Width: UCS-2 represents each character strictly as a single 16-bit code unit, limiting native representation to the Basic Multilingual Plane (BMP, code points U+0000 through U+FFFF).
  • Surrogate Pairs: While modern EDK II implementations increasingly handle UTF-16 surrogate pairs without crashing, the core UEFI specification treats surrogate pairs as two distinct UCS-2 code units rather than a single compound scalar value.
  • Byte Ordering: Strings follow the host architecture’s native endianness, which is little-endian on standard x86, ARM, and RISC-V platforms.

Memory Representation and Termination

Internally, a UEFI Unicode string is managed as an array of 16-bit integers stored in contiguous memory:

  • Null Termination: Strings are null-terminated using a 16-bit null value (0x0000), requiring two zero bytes instead of the single zero byte used in standard 8-bit ASCII or C strings.
  • Alignment: Because CHAR16 is a 16-bit type, string buffers must be 2-byte aligned in system memory. Accessing an unaligned CHAR16 pointer can cause alignment faults on architectures with strict memory access requirements (such as certain ARM configurations).

UEFI Unicode Collation Protocol

High-level string manipulation—such as locale-aware case folding, wild-card matching, and comparison—is delegated to the EFI_UNICODE_COLLATION_PROTOCOL (or the newer EFI_UNICODE_COLLATION2_PROTOCOL, which supports RFC 4646 language tags).

The protocol exposes several function pointers that manage strings internally:

  • StriColl (String Collate): Performs locale-sensitive, case-insensitive string comparisons. Internally, the driver normalizes characters using language-specific uppercase tables before comparing their numerical values.
  • MetaiMatch: Implements pattern matching using wildcard characters (*, ?, and [ ]). The service evaluates UCS-2 characters against pattern tokens recursively or iteratively through the buffer.
  • StrLwr and StrUpr: Mutate strings in-place by looking up each CHAR16 value in an internal conversion lookup table defined by the active language (e.g., English "en" vs. French "fr").
  • FatToStr and StrToFat: Handle conversions between UCS-2 and the 8.3 short-filename format used by the FAT file system, substituting illegal characters with valid fallback symbols.

Base Library (EDK II) String Services

For raw string manipulation that does not require localized collation, UEFI core implementations (such as EDK II) provide lightweight, fast-path functions via BaseLib:

  • Linear Scanning: Functions like StrLen, StrCpy, and StrCmp iterate through memory two bytes at a time until encountering 0x0000.
  • Lexicographical Comparison: Standard StrCmp operations compare raw 16-bit integer values rather than linguistic weights, making them deterministic and independent of firmware localization settings.
  • Safe String Functions: Modern EDK II includes bounds-checking variants (e.g., StrCpyS, StrCatS) that accept a maximum character count to prevent buffer overflows during firmware execution.

ASCII and UTF-8 Conversions

Firmware often interfaces with external components (such as hardware devices, ACPI tables, and NVRAM variables) that produce or consume 8-bit strings. UEFI string services handle this via internal conversion routines:

  • Narrowing (Unicode to ASCII): Truncates the upper 8 bits of each CHAR16 unit. If a character resides outside the ASCII range (> 0x007F), it is replaced with an underscore (_) or an implementation-defined placeholder.
  • Widening (ASCII to Unicode): Zero-extends standard 8-bit CHAR8 values into 16-bit CHAR16 units by padding the high-order byte with 0x00.