Why Standard TLS Cannot Be Used Directly Over UDP

Standard Transport Layer Security (TLS) cannot be used directly over User Datagram Protocol (UDP) because TLS fundamentally relies on the reliability, ordering, and stream-oriented delivery provided by TCP. UDP is an inherently connectionless and unreliable protocol that permits packet loss, reordering, and duplication. Because standard TLS lacks built-in mechanisms to handle these network anomalies, running it over UDP causes cryptographic state desynchronization, broken handshakes, and immediate connection failures, which is why Datagram Transport Layer Security (DTLS) was created as a distinct alternative.

The Requirement for Reliable Handshakes

The standard TLS handshake is a strictly sequential exchange of cryptographic parameters, certificates, and keys. Each step relies entirely on the successful arrival and verification of the previous step. Because standard TLS assumes a reliable transport layer like TCP, it does not include its own retransmission timers or acknowledgment mechanisms. If a single handshake packet is dropped over a UDP connection, standard TLS cannot recover and the connection simply stalls indefinitely.

Cryptographic State and Packet Reordering

TLS protects against replay attacks and verifies message integrity by using implicit sequence numbers. Each TLS record implicitly increments a counter that is factored into the Message Authentication Code (MAC) or Authenticated Encryption with Associated Data (AEAD) calculation.

If UDP delivers packets out of order, the receiver’s sequence counter falls out of sync with the sender’s. When this happens: * The decryption or authentication check fails. * TLS treats the authentication failure as a fatal security violation. * The TLS session is immediately terminated.

Lack of Native Fragmentation Handling

TLS produces records that can be up to 16 KB in size, which often exceeds the Maximum Transmission Unit (MTU) of network paths. Over TCP, large records are smoothly divided into smaller segments. Over UDP, records that exceed the MTU must rely on IP fragmentation, which is frequently blocked or dropped by intermediate network routers and firewalls. Standard TLS has no record-layer fragmentation mechanism to break down security payloads independently of the underlying transport.

The Solution: DTLS

To address these fundamental incompatibilities, Datagram Transport Layer Security (DTLS) was developed. DTLS adapts the standard TLS protocol specifically for datagram delivery by introducing: * Explicit Sequence Numbers: Included in each record header so out-of-order packets can be decrypted independently. * Retransmission Timers and Handshake Sequence Numbers: To guarantee the handshake completes even if packets are lost. * Replay Detection Windows: Utilizing a sliding bitmask to detect replayed packets without strictly requiring in-order arrival.