Multibyte to Wide Character Conversion in C

Converting between multibyte strings (char*) and wide-character strings (wchar_t*) in the C programming language is essential for handling internationalized text. The standard library provides mbstowcs and wcstombs declared in <stdlib.h> to perform these conversions. This article explains how both functions work, how the system locale influences encoding, the function signatures, return values, and how to safely size destination buffers.

Core Concepts: Multibyte vs. Wide Characters

  • Multibyte characters (char): Variable-length encoding sequences (such as UTF-8 or Shift-JIS) where a single visual character can span from one to several bytes.
  • Wide characters (wchar_t): Fixed-width characters (typically 16-bit on Windows or 32-bit on Linux/POSIX systems) designed to hold any character of the system's execution character set in a single element.

Because the byte representation varies across character sets, conversion depends entirely on the current locale setting. Programs must initialize the locale using setlocale(LC_CTYPE, "") or setlocale(LC_ALL, "") before calling conversion functions; otherwise, the default "C" locale is used, which typically restricts characters to 7-bit ASCII.

Converting Multibyte to Wide Characters: mbstowcs

The function mbstowcs (multibyte string to wide-character string) converts an array of multibyte characters into an array of wide characters.

Prototype

size_t mbstowcs(wchar_t *dest, const char *src, size_t max);

Parameters

  • dest: Pointer to the destination wide-character buffer.
  • src: Pointer to the null-terminated source multibyte string.
  • max: Maximum number of wchar_t elements to write into dest.

Behavior and Return Value

  • Success: Returns the number of wide characters written to dest, excluding the terminating null wide character (L'\0').
  • Invalid Sequence: Returns (size_t)-1 if an invalid multibyte character is encountered under the active locale.
  • Buffer Sizing: If dest is NULL, mbstowcs returns the required buffer capacity in wchar_t units (excluding the terminating null character) without writing anything.

Converting Wide Characters to Multibyte: wcstombs

The function wcstombs (wide-character string to multibyte string) converts an array of wide characters back into a multibyte character sequence.

Prototype

size_t wcstombs(char *dest, const wchar_t *src, size_t max);

Parameters

  • dest: Pointer to the destination byte buffer.
  • src: Pointer to the null-terminated source wide-character string.
  • max: Maximum number of bytes to write into dest.

Behavior and Return Value

  • Success: Returns the total number of bytes written to dest, excluding the terminating null byte ('\0').
  • Unrepresentable Character: Returns (size_t)-1 if a wide character cannot be mapped to a valid multibyte sequence in the current locale.
  • Buffer Sizing: If dest is NULL, wcstombs calculates the required byte size (excluding the terminating null byte) needed to store the converted string.

Practical Example

#include <stdio.h>
#include <stdlib.h>
#include <locale.h>

int main(void) {
    // 1. Enable native locale (e.g., UTF-8)
    setlocale(LC_ALL, "");

    const char *mb_input = "Hello, 世界!"; // Multibyte UTF-8 string

    // 2. Measure required wchar_t buffer length
    size_t wlen = mbstowcs(NULL, mb_input, 0);
    if (wlen == (size_t)-1) {
        perror("mbstowcs sizing error");
        return 1;
    }

    // 3. Allocate and convert to wide characters
    wchar_t *wide_str = malloc((wlen + 1) * sizeof(wchar_t));
    mbstowcs(wide_str, mb_input, wlen + 1);

    // 4. Measure required char buffer length for conversion back
    size_t mblen = wcstombs(NULL, wide_str, 0);
    if (mblen == (size_t)-1) {
        perror("wcstombs sizing error");
        free(wide_str);
        return 1;
    }

    // 5. Allocate and convert back to multibyte
    char *mb_output = malloc(mblen + 1);
    wcstombs(mb_output, wide_str, mblen + 1);

    printf("Original:  %s\n", mb_input);
    printf("Roundtrip: %s\n", mb_output);

    free(wide_str);
    free(mb_output);
    return 0;
}

Important Considerations

  • Null Termination: If max is reached before the terminating null character of the source is encountered, the destination string will not be null-terminated.
  • Shift States: mbstowcs and wcstombs always begin conversion in the initial shift state. If processing strings with shift states in chunks or streams, the restartable variants mbsrtowcs and wcsrtombs should be used instead.
  • Platform Differences: The size of wchar_t is implementation-defined (2 bytes on Windows using UTF-16, 4 bytes on most Unix-like platforms using UTF-32), meaning wide-character buffers behave differently across operating systems.