Protocol Buffers and FlatBuffers in C Systems

Protocol Buffers and FlatBuffers are high-performance binary serialization formats designed to exchange structured data efficiently across networks and storage. This article explores the architecture of both formats, compares their internal serialization models, and explains how to integrate them into resource-constrained and performance-critical systems using the C programming language.

Understanding Protocol Buffers and FlatBuffers

Both Protocol Buffers (Protobuf) and FlatBuffers define data schemas in platform-neutral .proto or .fbs files, which a dedicated compiler converts into native source code. Despite sharing this workflow, their data representation and runtime execution differ fundamentally.

Protocol Buffers (Protobuf)

Developed by Google, Protobuf focuses on minimizing payload size. It serializes data by encoding fields using variable-length integers (varints) and tag-length-value representations. To consume a Protobuf message, an application must deserialize the raw binary stream: the parser traverses the entire buffer, decodes the values, and allocates memory to construct an in-memory object graph.

FlatBuffers

Also developed by Google, FlatBuffers focuses on zero-copy access and execution speed. Data is stored in internal buffers using specific byte alignments and internal offset tables (vtables). Because the layout in the serialized buffer mirrors the layout required by memory-mapped structures, an application can read fields directly from the raw byte buffer without a decoding or memory allocation phase.

Protobuf vs. FlatBuffers: Core Architectural Differences

Feature Protocol Buffers FlatBuffers
Parsing Step Required (Unpacking bytes to native structs) None (Zero-copy, read directly from wire format)
Wire Size Highly compact (uses varints, bit packing) Slightly larger (requires padding for byte alignment)
Memory Allocation Typically requires heap allocations for unpacked objects Operates directly on the input buffer; minimal to no allocation
Read Latency Dependent on total payload size and parsing speed Near-instantaneous random access via offset indexing

Implementation in the C Programming Language

The official Google implementations for both libraries are written in C++. Using these formats in pure C—common in Linux kernel modules, microcontrollers, real-time operating systems (RTOS), and embedded devices—requires dedicated C-native libraries.

Implementing Protocol Buffers in C

For C applications, the most common implementations are protobuf-c and nanopb.

  • protobuf-c: A full C implementation that mirrors the official C++ API. It maps messages to C structs and uses descriptors for reflection. It relies on standard memory allocators (malloc/free), making it suitable for desktop or server-grade C applications.
  • nanopb: Engineered specifically for memory-constrained embedded systems. It compiles .proto files into static C structs and generates deterministic encoding and decoding callbacks. Nanopb allows fixed-size buffers, completely eliminating the need for dynamic memory allocation on the heap.

A typical workflow with nanopb involves:

  1. Compiling the .proto file using nanopb_generator.py to create .pb.c and .pb.h files.
  2. Initializing a native C struct matching the generated type.
  3. Passing an output byte stream (pb_ostream_t) and the struct to pb_encode() to produce the binary payload.
  4. Parsing an incoming message using pb_istream_t and pb_decode() to populate the C struct.

Implementing FlatBuffers in C

The primary tool for C integration is flatcc (FlatBuffers in C). It provides an optimizing schema compiler and runtime library that produces portable C99 code.

flatcc provides two distinct modes:

  • Reader: A set of lightweight macros and inline functions that read directly from the buffer. It requires no memory allocation, does not modify the source buffer, and handles endianness conversions transparently.
  • Builder: A stack-based state machine that constructs FlatBuffer payloads from the bottom up, emitting properly aligned binary tables.

A typical workflow with flatcc involves:

  1. Compiling the schema (flatcc -a schema.fbs) to generate header files.
  2. Creating an emitter using flatcc_builder_t.
  3. Constructing nested tables and vectors using emitted macros (e.g., MyTable_start(), MyTable_add_field(), MyTable_end()).
  4. Accessing fields on the receiving end directly via accessors (e.g., MyTable_field_get(table_pointer)), reading directly from the serialized pointer.

Selection Criteria for C Systems

Choose Protocol Buffers when network bandwidth or disk storage is the primary constraint, or when integrating with an existing infrastructure that heavily relies on Protobuf schemas. Libraries like nanopb make it practical for microcontrollers where payloads must remain as small as possible over low-throughput links like CAN, BLE, or LoRa.

Choose FlatBuffers when low latency, deterministic execution, and zero-allocation constraints dominate. Because it bypasses the parse-and-allocate phase, it is suited for high-frequency trading engines, shared-memory inter-process communication (IPC), device drivers, and real-time gaming or multimedia pipelines written in C.