To see how Protobuf gets its performance, we need to look at the raw bytes; this is the physical layer of the specification.
Throughout this guide, we'll use Protoscope, a specialized human-readable language that visualizes binary Protobuf streams. It's an excellent tool for bridging the gap between raw hexadecimal bytes and structured data.
Binary 101
Foundational concepts like Endianness and Bit Shifting.
Base-128 Varints
Variable-length integers for efficient packing.
ZigZag
How signed integers are compressed.
The Field Tag
How field numbers and wire types are packed.
Wire Types
Envelopes, packed fields, and sub-messages.
Map Encoding
How compound types are represented.
Presence
Why empty fields take zero bytes.
Parse Loop
How each field is read until bytes run out.
Binary Explorer
Live inspection and disassembly.
This guide should help explain how the Protobuf encoding works, but if you have questions about edge cases or specifics that we don't cover, refer to the official encoding guide.
Before we dig into Protobuf encoding, there are three concepts to cover when working with binary data: Endianness, bit shifting, and the bitwise OR operator.
Endianness
Endianness refers to the order in which bytes are arranged in computer memory. Systems generally use either Big-Endian (most significant byte first) or Little-Endian (least significant byte first).Protobuf uses a Little-Endian approach for most integer types.
In Little-Endian systems, the "least significant byte" (the one with the smallest numerical weight) is stored first in the sequence. When we look at Protobuf's raw binary streams, this explains why smaller pieces of a number often appear before larger pieces.
Bit Shifting
Bitwise shifting is the primary tool for Bit Packing. It allows us to manipulate individual bits within a byte, moving them to "make room" for other data.
A left shift (<<) moves every bit in a number to the left. The empty spaces on the right become zeroes. Think of it like adding a zero to the end of a base-10 number (which multiplies it by 10). In binary, shifting left by 1 multiplies the number by 2. Shifting left by 3 multiplies it by 8 (23), and creates exactly 3 empty bits on the right side.
00000001
<< 3
= 00001000Bit shifting is a foundational operation for data serialization. By shifting bits to the left, we can "make room" on the right side. Protobuf uses this exact technique: it shifts the field number to the left by 3, creating 3 empty bits to store the wire type.
The OR Operation
The bitwise OR (|) is used to combine binary values. It acts like a logic gate: if either input bit is 1, the output bit is 1.
Once we've shifted bits to make room (like creating 3 empty zeros), we can use the OR operation to "glue" another value into that space.
00011000 | 00001010 = 00011010
The bitwise OR operation allows us to safely combine separate values into a single number. As long as the individual values are shifted to occupy non-overlapping bit ranges (like a puzzle fitting together), they can be merged without corrupting one another.
Varints are the fundamental building block of Protobuf efficiency, allowing integers to occupy only as many bytes as necessary.
Standard integers in memory take 4 or 8 bytes regardless of their value. Varints use Base-128 Serialization to represent smaller numbers with fewer bytes.
Each byte in a varint, except the last byte, has the most significant bit (MSB) set to 1. This acts as a continuation flag, telling the decoder "more bytes are coming."
The lower 7 bits of each byte store the data in groups of 7, least significant group first. This means Protobuf uses a Little-Endian approach even at the bit-group level.
Chunk Data
Split the number into 7-bit groups. Standard bytes are 8 bits, but we reserve the top bit (MSB) as a "continuation bit".
Reverse & Add MSB Flag
The groups are written in Little-Endian order (least significant group first). Set the MSB to 1 for all bytes except the last one.
Standard Varints are great for positive numbers, but they are highly inefficient for negative ones. ZigZag encoding fixes this.
Standard Varint (Two's Complement)
10 Byte(s)💡 Silliness Warning: Because of Two's Complement, a tiny negative number like -1 requires all 10 bytes (80 bits) under standard varint encoding, while positive 1 requires only 1 byte! This is a massive 10x space overhead.
ZigZag Varint
1 Byte(s)💡 ZigZag Scaling: Space usage is determined strictly by the absolute magnitude of the number rather than its sign. The greater the magnitude, the more bytes will be used. A small negative number like -1 takes only 1 byte!
In traditional Two's Complement encoding, the sign of a number is stored in the Most Significant Bit (MSB). Because Varints strip leading zeros, a negative number (which has a leading 1 in the MSB) is forced to act like a very large unsigned integer, requiring all 10 bytes of a 64-bit varint just to represent -1.
ZigZag encoding solves this by moving the sign bit to the Least Significant Bit (LSB). It maps positive numbers to even integers (n << 1) and negative numbers to odd integers.
By pushing the sign bit to the bottom and stripping the empty leading zeroes, Protobuf ensures that small negative numbers take just as little space as small positive numbers.
This is named ZigZag because the mapping zig-zags back and forth between positive and negative integers as you count up the encoded values: 0 maps to 0, -1 to 1, 1 to 2, -2 to 3, 2 to 4, and so on.
Every field in a Protobuf message is prefixed by a Tag. This tag is the only reason the decoder knows which field it's currently processing and how to interpret the bytes that follow.
Tag Composition
A tag is a single Varint that combines two pieces of information:
- Field Number (bits 3 through N)
- Wire Type (the bottom 3 bits)
The formula for the tag value is (field_number << 3) | wire_type. While small field numbers fit in a single byte, larger ones (16 and above) will trigger the Varint continuation logic and require additional bytes.
This visualization shows how Field Number 1 and Wire Type 2 (Varint) are combined into a single tag value of 10, which is then encoded as a Base-128 Varint.
01. Bit Shift (field << 3)
02. Bitwise OR (| wire_type)
03. Varint Encoding (Little-Endian)
Every field on the wire is wrapped in an "envelope" that tells the decoder two things: which field number it is, and how to read the payload. These are packed into a single Tag byte.
Varint
Type 0 (000)Most numeric types. Variable length allows small numbers to take very little space.
I64
Type 1 (001)Fixed 64-bit values. Used when values are frequently large or need exact precision.
LEN
Type 2 (010)Length-delimited blobs. Includes a length prefix followed by the payload data.
I32
Type 5 (101)Fixed 32-bit values. Efficient for common hardware types.
Varint Fields (Wire Type 0)
Most numeric types use Wire Type 0. The length is implicit because the decoder reads bytes one by one until it finds a byte where the MSB (Most Significant Bit) is 0. This "continuation bit" logic allows the payload to be self-delimiting.
1: 150 // Field 1 (Varint) with value 150Fixed-Size Fields (Wire Type 1, 5)
For double, fixed64 (Wire Type 1), and float, fixed32 (Wire Type 5), the length is hard-coded into the specification. The decoder knows to read exactly 8 or 4 bytes respectively immediately following the tag.
// Field 2: float (value 3.14) 2: 0x40490fdb // Field 3: double (value 80.0) 3: 0x4054000000000000
Length-Delimited (Wire Type 2)
Strings, bytes, and nested messages use Length-Delimited encoding. These fields include an explicit length byte (encoded as a varint) immediately after the tag, telling the decoder exactly how many subsequent bytes belong to this field.
3: {"Alice"} // Field 3 (Length-delimited string)Packed Repeated Fields
Repeated fields of primitive types use a specialized encoding to avoid repeating the field tag for every element.
Each element repeats the field tag. High overhead for many small elements.
Elements are concatenated into a single length-delimited record. One tag for the whole set.
Maps are not a native wire-level primitive. Instead, they are syntactic sugar for a repeated message.
map<string, int32> items = 1;
message Entry { string key = 1; int32 value = 2; } repeated Entry items = 1;
One of the biggest space savers in Protobuf is the omission of default values.
Standard fields are omitted from the wire if they hold the default value.
Optional fields track explicit presence. They are serialized even if set to 0.
In Proto3, fields set to their default value (0, empty string, false) are not serialized at all. This makes the wire format extremely compact, but it means you cannot distinguish between "set to 0" and "not set."
The optional keyword (and oneof) reintroduces Explicit Presence. These fields are wrapped in a way that ensures they are written to the wire even if their value is the default, allowing for has_field() checks.
Note on Editions: With the introduction of Protobuf Editions, the strict boundaries between Proto2 and Proto3 behavior have been removed. You can now explicitly configure whether fields use implicit or explicit presence via features like features.field_presence = EXPLICIT;, giving you granular control over serialization size versus field state tracking.
When multiple fields are sent together, Protobuf concatenates them into one binary stream. The decoder does not need separators between fields; each field tells the decoder how many bytes to consume before moving on.
Read the tag
The first varint in each field is the tag. Its lower three bits identify the wire type, and the remaining bits identify the field number.
Choose the payload rule
The wire type tells the decoder how to find the payload boundary: varint continuation bits, a fixed 4 or 8 byte width, or a length-delimited size prefix.
Consume that field
Once the payload length is known, the decoder consumes exactly those bytes, maps them to the schema field when possible, and advances its cursor.
Repeat until EOF
The stream has no outer field count. Parsing continues from the next byte and stops only when there are no bytes left to process.
Try modifying the JSON data below or clicking the example buttons to see how the binary stream changes in real-time. Click any segment in the encoded stream to inspect how its tag, length, and payload were parsed.
The Wire-Level Language
Protoscope is a specialized tool and human-editable language designed for inspecting, debugging, and manually constructing Protocol Buffers wire format data.
Standard tools like protoc require a .proto schema to make sense of binary data. Protoscope is different: it operates at the wire level, decoding the underlying binary structure (varints, tags, and length-prefixes) using heuristics, even when the original schema is missing.
Strengths
- +
Schema-agnostic debugging of any Protobuf stream.
- +
Perfect for crafting malformed messages for security testing.
- +
Human-readable representation of complex binary structures.
Weaknesses
- -
Lossy: Cannot distinguish between
int32anduint32. - -
Ambiguous: May misidentify embedded messages as strings.
- -
No field names: You only see numeric tags (e.g.,
1:).
1. Heuristic Disassembly
Protoscope scans bytes and "guesses" types based on valid UTF-8 sequences or nested tag patterns.
2. Minimalist Syntax
3. Bi-directional
It can compile text back into binary, making it a powerful "hex editor" for Protobuf.
Try modifying the JSON data below or clicking the example buttons to see how the Protoscope output changes in real-time.
Payload Input
Protoscope Output
ProtoscopeEnd of Transmission
This concludes our dive into the binary format. You should now have a solid intuition for how Protobuf packs data into raw bytes, how varints work, and how the tag system structures the stream.
For exhaustive details, edge cases, and the formal specification, always refer to the official Protobuf Encoding Guide.