How strtok_r Maintains State in C

The strtok_r function is the reentrant and thread-safe equivalent of the standard C library function strtok. While standard strtok tracks its position in a string using a hidden, shared static buffer—causing race conditions in multithreaded programs—strtok_r avoids this by offloading the state-tracking responsibility to the caller. This article explains how strtok_r leverages an external pointer parameter to maintain parsing context reliably across multiple invocations without global state.

The Mechanism of saveptr

The core difference between strtok and strtok_r lies in the function signatures. The prototype for strtok_r is:

char *strtok_r(char *str, const char *delim, char **saveptr);

The third argument, saveptr, is a pointer to a char * variable. This user-provided variable acts as the storage container for the function's internal state. Because the caller allocates this variable—typically as a local variable on the thread's execution stack—each thread or parsing task maintains its own isolated state context.

Step-by-Step Execution Lifecycle

The parsing process with strtok_r follows a specific protocol across consecutive calls:

1. Initial Call: On the first invocation, the caller supplies the address of the target string in str and the address of the tracking pointer in saveptr (e.g., &saveptr). The function scans str to locate the first token, skips any leading delimiters defined in delim, and determines the end of the token. It replaces the first delimiter character following the token with a null terminator (\0) and updates the memory location pointed to by saveptr to store the address of the next character in the buffer. The function then returns a pointer to the start of the found token.

2. Subsequent Calls: For subsequent tokens from the same string, the caller passes NULL as the str argument while continuing to pass the same saveptr address. Detecting that str is NULL, strtok_r reads the address stored inside *saveptr as its starting position rather than starting over. It repeats the process of scanning, null-terminating, advancing *saveptr, and returning the next token.

3. Termination: Once the end of the string is reached and no more tokens remain, strtok_r updates *saveptr to point to the end of the buffer (or sets it to NULL) and returns NULL. This signals to the calling loop that tokenization is complete.

Code Example

Below is a standard pattern demonstrating how the caller manages the parsing state:

#include <stdio.h>
#include <string.h>

int main(void) {
    char input[] = "apple,banana,cherry";
    const char delim[] = ",";
    char *saveptr;

    // First call: pass the string to tokenize
    char *token = strtok_r(input, delim, &saveptr);

    // Subsequent calls: pass NULL to continue using saveptr
    while (token != NULL) {
        printf("Token: %s\n", token);
        token = strtok_r(NULL, delim, &saveptr);
    }

    return 0;
}

Why This Design Ensures Thread Safety

Because standard strtok uses an internal static char * variable, concurrent calls from different threads overwrite each other's progress through their respective strings. By requiring the user to supply saveptr, strtok_r removes all global and static state from the function itself. Multiple threads can safely execute strtok_r simultaneously, provided each thread supplies its own distinct char * tracking variable.