How UTF-8 Encodes Unicode Code Points in Binary

UTF-8 is a variable-length character encoding system that translates Unicode code points into one to four 8-bit bytes. By using specific binary prefixes for each byte, UTF-8 preserves backward compatibility with standard 7-bit ASCII, prevents byte-synchronization errors, and efficiently handles the entire Unicode character space ranging from U+0000 to U+10FFFF.

The UTF-8 Binary Mapping Template

Unicode assigns every character a unique numerical value known as a code point, denoted as U+XXXX in hexadecimal format. UTF-8 divides these code points into four distinct numerical ranges. Each range corresponds to a fixed byte length determined by predefined binary prefix bits:

The x positions represent the actual binary bits of the Unicode code point (payload), while the non-x bits serve as structural markers governed by UTF-8 rules.

Prefix Rules and Byte Roles

The binary prefixes establish the role of each byte in a sequence:

• Single-Byte Characters (0xxxxxxx): A leading bit of 0 signals a 1-byte sequence, matching standard ASCII character encoding identically.
• Leading Bytes (110..., 1110..., 11110...): In a multi-byte sequence, the first byte indicates the total sequence length. The number of consecutive 1s before the first 0 denotes how many bytes comprise the character.
• Continuation Bytes (10xxxxxx): Every subsequent byte in a multi-byte sequence begins with the two-bit prefix 10. This distinction guarantees that a continuation byte can never be mistaken for a leading byte or an ASCII character.

The Mapping Process: Step-by-Step

To map any Unicode code point to UTF-8 binary, follow these steps:

1. Convert to Binary: Express the hexadecimal code point as a raw binary integer.
2. Select the Byte Structure: Determine which of the four ranges contains the code point.
3. Distribute Bits: Insert the binary digits into the x slots of the selected template from right to left (least significant bit to most significant bit).
4. Pad Remaining Slots: Fill any remaining unoccupied x bits on the far left of the payload with 0s.

Example: Encoding the Euro Sign (€ / U+20AC)

1. Hex to Binary: U+20AC converted to binary is 0010 0000 1010 1100.
2. Determine Range: U+20AC falls between U+0800 and U+FFFF, requiring a 3-byte template:
   1110xxxx 10xxxxxx 10xxxxxx
3. Fill the Template:
   • Payload: 0010 0000 1010 1100 (16 bits)
   • Byte 1: 1110 + 0010 = 11100010 (0xE2)
   • Byte 2: 10 + 000010 = 10000010 (0x82)
   • Byte 3: 10 + 101100 = 10101100 (0xAC)
4. Result: The UTF-8 binary representation is 11100010 10000010 10101100 (Hex: E2 82 AC).

Benefits of Binary Prefix Encoding

This structural design ensures that no valid UTF-8 sequence forms a subsequence of another valid sequence. Decoders can start reading at any random point in a byte stream and instantly identify whether a byte is a single-byte character, the beginning of a multi-byte character, or a continuation byte, enabling fast error recovery and stream resynchronization.