How strtok Works and Why It Is Thread-Unsafe in C

The strtok function in the C standard library is a utility used to split a string into a sequence of tokens based on specified delimiter characters. It functions by sequentially scanning the input string, replacing delimiter characters with null terminators ('\0'), and returning pointers to the beginning of each identified token. However, because strtok relies on a hidden static pointer to maintain its position between successive calls, it is inherently thread-unsafe, leading to race conditions and data corruption when accessed concurrently by multiple threads.

Algorithmic Function of strtok

The strtok function is declared as char *strtok(char *str, const char *delim). It operates via a stateful, iterative mechanism:

  1. Initialization: On the initial call, the caller passes the target string via str and a set of delimiters via delim. On subsequent calls to retrieve further tokens from the same string, str must be passed as NULL.
  2. Skipping Leading Delimiters: The function scans the string starting from the current position, skipping any characters that match any character in the delim set. If it encounters the end of the string ('\0') without finding a non-delimiter character, it returns NULL.
  3. Locating the Token Boundary: Once the start of a token is found, strtok advances through the string until it encounters the next character that matches any delimiter in delim or reaches the string's null terminator.
  4. String Modification: If a delimiter is found, strtok overwrites that delimiter character in memory with a null terminator ('\0'). This effectively terminates the current token as a valid, stand-alone C string.
  5. State Preservation: The function stores the memory address of the character immediately following the newly placed null terminator in an internal, static pointer.
  6. Return Value: It returns a pointer to the beginning of the extracted token. If no token could be found, it returns NULL.

Because strtok modifies the string in place by inserting null terminators, the original string is permanently altered. Consequently, passing a string literal (which resides in read-only memory) results in undefined behavior, typically a segmentation fault.

Why strtok Is Inherently Thread-Unsafe

The thread-safety issue in strtok stems directly from its requirement to preserve state between function invocations without taking a state parameter from the caller.

To track where the next search should begin when str is NULL, standard implementations of strtok store the continuation address in an internal variable with static storage duration:

static char *last_position;

Because variables with static storage duration share a single memory location across the entire process:

  • Shared State Across Threads: If Thread A calls strtok(strA, delim) to begin parsing a string, the internal pointer is set to a location within strA.
  • Race Conditions: If Thread B calls strtok(strB, delim) before Thread A completes its tokenization loop, the shared internal pointer is overwritten with a location inside strB.
  • Corrupted Execution: When Thread A subsequently calls strtok(NULL, delim), it will not resume parsing strA. Instead, it reads the pointer modified by Thread B, leading to interleaved token streams, skipped data, memory access violations, or infinite loops.

Thread-Safe Alternatives

To address this limitation, modern standards provide reentrant alternatives that eliminate the shared static state:

  • strtok_r (POSIX): Accepts a third argument, char **saveptr, which is a pointer to a caller-allocated char* variable. The caller maintains the state on their local stack, isolating each thread's execution context.
  • strtok_s (C11 Bounds-checking Annex K): Functions similarly to strtok_r, requiring a caller-provided context pointer to track progress across calls safely.