Protobuf offers advanced features for modeling richer APIs and evolving schemas safely over time. Descriptor-based extension, reflection, and validation workflows now have dedicated pages.
Modeling Track
Imports, dynamic payloads, field masks, and services.
Evolution Track
Compatibility, editions, presence, and limits.
Imports
Managing dependencies and resolving import paths.
Version Editions
The modern evolution of Protobuf feature management.
Services
Defining RPC interfaces for networked communication.
Standard Options
Built-in metadata for generated code and runtime behavior.
Breaking Changes
Compatibility checks for evolving schemas safely.
Presence
Distinguishing between default values and missing data.
Required Fields
Handling 'required' fields and business rules in an evolvable way.
Size Limits
Architectural constraints and memory behavior.
03_TRACK_A
Schema Modeling
These sections cover the schema features used to model larger APIs: imports, dynamic payloads, partial updates, and service definitions.
You can use definitions from other .proto files using the import statement. Imports from another team or project are the awkward part: the traditional answer is to copy their files into your repo and re-copy them whenever they change.
The Buf Schema Registry treats them as versioned dependencies instead, the way NPM or Cargo would. You name a module under deps in buf.yaml, buf dep update pins exact versions in buf.lock, and the imports resolve identically for everyone who checks out the repo. The schema you are editing on this site does exactly that with buf.build/bufbuild/protovalidate.
Always import using fully qualified paths from your module root to avoid baffling "Duplicate Symbol" errors.
edition = "2023"; package common.v1; message User { string id = 1; string name = 2; }
edition = "2023"; package auth.v1; import "common/v1/user.proto"; message LoginResponse { common.v1.User user = 1; string session_token = 2; }
version: v2 modules: - path: proto # Registry modules, resolved on demand. Nothing vendored. deps: - buf.build/bufbuild/protovalidate - buf.build/googleapis/googleapis
# Pin every dep to an exact version in buf.lock $ buf dep update # Imports resolve relative to your buf.yaml root $ buf build
The Any type allows you to include messages where the schema isn't known at compile time.
google.protobuf.Any embeds an arbitrary serialized Protobuf message along with a URL that identifies its type (e.g., type.googleapis.com/mypackage.MyMessage).
When serialized to ProtoJSON, this type identifier is rendered as a special @type property alongside the standard JSON fields of the embedded message, allowing parsers to route the payload correctly.
// In Proto: import "google/protobuf/any.proto"; message Event { google.protobuf.Any payload = 1; } // In ProtoJSON: // { // "payload": { // "@type": "type.googleapis.com/demo.User", // "name": "Hiro" // } // }
If you are working with dynamic protobuf messages, use Any. However, if you are working with arbitrary structured JSON data that we don't want to model or is completely dynamic (like a schema-less JSON object), use google.protobuf.Value or google.protobuf.Struct.
A Value represents a dynamically typed value which can be either a null, a number, a string, a boolean, a recursive struct (object), or a list of values. It perfectly maps to any valid JSON structure.
Use this sparingly, as it defeats the purpose of Protobuf's strong typing, but it's useful for integrating with schemaless NoSQL databases or passing untyped metadata blocks.
// In Proto: import "google/protobuf/struct.proto"; message Event { // Represents any arbitrary JSON value google.protobuf.Value metadata = 1; // Represents specifically a JSON object google.protobuf.Struct custom_attributes = 2; } // In ProtoJSON: // { // "metadata": "simple string or object", // "custom_attributes": { // "dynamic_key": [1, 2, 3], // "enabled": true // } // }
google.protobuf.FieldMask is a well-known type used to identify a subset of fields in a request.
It is the standard way to express partial updates (PATCH), allowing a client to send only the modified fields instead of the entire object.
FieldMask also works for tuning read responses. You can design a single List or Get response that supports many optional fields and associations (e.g., user.profile, user.settings). The client passes a read_mask to tell the server exactly which subset of data to return, eliminating "over-fetching" without needing multiple specialized endpoints.
import "google/protobuf/field_mask.proto"; message GetUserRequest { string id = 1; // Client requests only specific fields // e.g. ["name", "email", "metadata.last_login"] google.protobuf.FieldMask read_mask = 2; } message UpdateUserRequest { User user = 1; // Client identifies which fields to update google.protobuf.FieldMask update_mask = 2; }
The service keyword is used to define RPC (Remote Procedure Call) interfaces. Frameworks like gRPC or ConnectRPC use these definitions to generate client and server code.
Services support four types of communication:
- Unary: Simple request-response.
- Streaming: Send or receive sequences of messages in a single call (Client, Server, or Bidirectional).
Note: While Protobuf provides the language to define these interfaces, the underlying networking protocols and implementation frameworks (like gRPC or ConnectRPC) are a broad topic and are out of scope for this guide.
service UserService { // Unary: One request, one response rpc GetUser(GetUserRequest) returns (User); // Server Stream: One request, many responses rpc ListUsers(ListUsersRequest) returns (stream User); // Bidirectional Stream: Real-time chat rpc Chat(stream Message) returns (stream Message); }
Protobuf options control how code is generated and how data is mapped. They are categorized by scope: File, Message, Field, or Service.
option go_package: Defines the Go import path.option java_package: Defines the Java package.option optimize_for = SPEED;: Generates highly optimized (but larger) code. Alternatives:CODE_SIZE,LITE_RUNTIME.[deprecated = true]: Marks a field as deprecated.[json_name = "custom"]: Sets a custom JSON key.
edition = "2023"; option go_package = "github.com/example/v1"; option java_multiple_files = true; option optimize_for = SPEED; message User { string user_id = 1 [json_name = "uid"]; string old_field = 2 [deprecated = true]; }
Not all breaking changes are equal. Tools like Buf categorize breaking changes into four distinct levels of severity.
- WIRE: The most severe level. This includes changing a field number or using an incompatible type (e.g.,
stringtoint32). This causes data corruption during serialization; you should never do this. - WIRE_JSON: Breakage in JSON representation. Renaming a field is safe on the binary wire, but clients expecting the old JSON key will fail. You can mitigate this using the
[json_name="old_name"]annotation. - PACKAGE: Source code breakage at the package level. Changing a type in a wire-compatible way (e.g.,
int32toint64) transmits safely, but when developers update their generated code, their builds will fail until they update their types. - FILE: The strictest level. This ensures source code compatibility down to the individual file level. Moving a message to another file might break code generation that relies on specific file imports.
edition = "2023"; package api.v1; message User { string id = 1; int32 age = 2; string display_name = 3; }
edition = "2023"; package api.v1; message User { // [WIRE] breakage: type changed from string int32 id = 1; // [PACKAGE] breakage: source code type change int64 age = 2; // [WIRE_JSON] breakage: JSON key changed string full_name = 3; }
03_TRACK_B
Schema Evolution
Compatibility is the difficult part of long-lived Protobuf systems. These sections focus on edition features and how presence affects API behavior.
Protobuf Editions unifies proto2 and proto3, allowing features to be toggled individually rather than through major syntax version upgrades.
Editions allows for smooth migrations and fine-grained control over behaviors:
- Field Presence: Choose between IMPLICIT (proto3 default) or EXPLICIT (proto2 default).
- Enum Type: OPEN enums allow unknown values, while CLOSED enums treat them as invalid.
- Repeated Encoding: Standardize on PACKED (for efficiency) or EXPANDED (for compatibility).
That granularity is the whole point. Under proto2 and proto3, these behaviors were welded to the single keyword at the top of the file, so migration was all or nothing: moving a file to proto3 opened its closed enums, dropped its custom field defaults, and removed explicit presence from its singular scalars, all at once.
Editions makes each of those a separate feature you can set per file, message, or field, so a schema can adopt one new behavior without taking the others. New behavior then ships as a new feature with a per-edition default, rather than as a "proto4" that would force the same all-or-nothing migration again.
edition = "2023"; // Globally enforce field presence option features.field_presence = EXPLICIT; message User { // Optional fields are back string name = 1; // Mixed behavior in one file! int32 age = 2 [features.enum_type = OPEN]; }
Implicit vs. Explicit
Field presence determines whether a receiver can distinguish between a field that was never set and one that was set to its default value (like 0 or ""). In short, implicit presence saves space by never sending default values, while explicit presenceincludes extra tracking to definitively tell you if a field was populated.
The Modern Solution
Due to widespread demand, the optional keyword was re-introduced in later versions of proto3 (v3.15+). Today, Protobuf Editions provides the clearest solution by allowing you to globally or locally toggle field_presence between IMPLICIT and EXPLICIT.
File-Level Default
edition = "2023"; // Set EXPLICIT presence for the entire file option features.field_presence = EXPLICIT; message Profile { string bio = 1; // Explicit (tracked) int32 views = 2; // Explicit (tracked) }
Field-Level Overrides
message LegacyData { // Override to IMPLICIT for specific fields int32 raw_id = 1 [features.field_presence = IMPLICIT]; // Follows file-level default (EXPLICIT) string note = 2; }
The Hard Limit
The absolute maximum size of a serialized protobuf message is 2 GiB. This is a hard architectural limit because the protocol relies on 32-bit signed integers to encode byte lengths and offsets. If a payload exceeds this size, standard parsers will throw an overflow error and refuse to read it.
The Typical Size
Protobuf is optimized for small, fast payloads. The official recommendation is to keep messages under a few megabytes. In practice, the ideal size is typically under 1 MB.
Once a message grows beyond 10 MB, the CPU and memory costs of parsing become highly noticeable. For moving large datasets, the standard pattern is to chunk the data into a stream of smaller messages.
Full Graph Parsing
Protobuf is fundamentally designed around the expectation that you will load the entire message into memory at once.
When you deserialize a payload, the parser reads the entire binary stream and instantiates a complete object graph.
In-Memory Expansion
As with most serialization formats, the resulting in-memory representation is significantly larger than the serialized binary. Pointers, object overhead, and data structure padding can cause memory usage to be several times the size of the original payload.