Schema Evolution Strategies for Production APIs

Production APIs are never finished. Business requirements shift, new features arrive, and legacy behaviors must be retired. The challenge is making these changes without disrupting the clients and services that already depend on the existing contract. Protocol Buffers were designed with this challenge in mind, but the wire format alone does not guarantee safe evolution. Teams must understand and follow a set of compatibility rules, test for regressions, and adopt disciplined lifecycle practices.

In the Nebula schema registry, schema evolution is a first-class concern. Every change to a .proto file passes through automated breaking-change detection before it can be merged. This article explains the principles behind that automation and the strategies that keep Nebula's inter-service contracts stable across dozens of independently deployed services.

Backward and Forward Compatibility

Two dimensions of compatibility matter in distributed systems. Backward compatibility means that new code can read data written by old code. Forward compatibility means that old code can read data written by new code. Protobuf's binary encoding supports both, provided certain rules are followed.

When a receiver encounters a field number it does not recognize, it preserves the bytes as an unknown field rather than failing. This is the mechanism that enables forward compatibility: an old consumer that has not been updated will simply ignore the new field, and if it re-serializes the message, the unknown field is carried through intact.

When a receiver expects a field that is absent from the incoming bytes, it fills in the default value for the type (zero for numbers, empty for strings and bytes, the zero-value enum variant). This is the mechanism for backward compatibility: a new consumer reading old data will see sensible defaults for fields that did not exist when the data was produced.

These guarantees hold as long as the schema changes respect the wire format. The following changes are safe:

// SAFE: Adding a new field
message PaymentEvent {
  string event_id = 1;
  string payment_id = 2;
  google.protobuf.Timestamp occurred_at = 3;
  // New field added in v1.3
  string correlation_id = 4;
}

Adding a new field with a previously unused field number is always safe. Old producers will not send it; old consumers will ignore it.

// SAFE: Renaming a field (wire format uses numbers, not names)
message PaymentEvent {
  string event_id = 1;
  string payment_id = 2;
  google.protobuf.Timestamp occurred_at = 3;
  string trace_id = 4;  // was correlation_id
}

Renaming a field does not affect the binary encoding. However, it does change generated code, so downstream consumers in languages that reference the field by name will need to update. In the Nebula registry, field renames are treated as a soft breaking change that requires coordinated rollout.

Dangerous Changes and How to Avoid Them

Several types of changes will silently corrupt data or cause deserialization failures. Understanding why they are dangerous is the first step toward avoiding them.

Changing a field's type is the most common source of silent corruption. Consider this scenario:

// Version 1
message Account {
  int32 balance = 1;
}
 
// Version 2 - DANGEROUS: changed type from int32 to string
message Account {
  string balance = 1;
}

The producer sends a string-encoded balance, but the consumer still expects an int32. The varint decoder will attempt to interpret UTF-8 bytes as a number, producing garbage. Protobuf's wire types will catch some mismatches (for example, varint vs. length-delimited), but others (int32 vs. uint32, which share the same wire type) will decode without error, yielding silently wrong values.

Reusing a deleted field number is equally dangerous:

// Version 1
message Order {
  string order_id = 1;
  string customer_name = 2;  // removed in version 2
  int32 quantity = 3;
}
 
// Version 2 - DANGEROUS: reused field number 2
message Order {
  string order_id = 1;
  int64 total_cents = 2;  // field 2 was customer_name!
  int32 quantity = 3;
}

An old producer still sends customer_name as field 2. The new consumer decodes those bytes as total_cents, producing a nonsensical integer. The correct approach is to use a reserved declaration:

message Order {
  reserved 2;
  reserved "customer_name";
 
  string order_id = 1;
  int32 quantity = 3;
  int64 total_cents = 4;  // new field with a new number
}

The reserved directive serves two purposes: it prevents the field number from being accidentally reused, and it prevents the old field name from being accidentally reintroduced. The Buf linter enforces both.

The Field Lifecycle: Add, Deprecate, Reserve

Every field in the Nebula registry follows a three-phase lifecycle.

Phase 1: Addition. A new field is added with a fresh field number. The pull request includes documentation comments explaining the field's purpose, format, and any constraints. If the field represents a replacement for an existing field, both fields coexist during a migration window.

message Transfer {
  string transfer_id = 1;
 
  // Deprecated: use structured_amount instead. Will be removed after 2025-06-01.
  int64 amount_cents = 2 [deprecated = true];
 
  // Structured monetary amount with currency.
  Money structured_amount = 5;
}

Phase 2: Deprecation. The old field is marked with [deprecated = true]. This annotation propagates into generated code, triggering compiler warnings in languages that support it. The deprecation comment includes a deadline, giving consuming teams a concrete migration window.

During this phase, producers send both the old and new fields. Consumers that have been updated read the new field; consumers that have not yet been updated continue reading the old field without disruption.

Phase 3: Reservation. After the migration window closes and monitoring confirms that no service is reading or writing the old field, it is removed from the message and its number and name are reserved. At this point, the field number can never be reused.

message Transfer {
  reserved 2;
  reserved "amount_cents";
 
  string transfer_id = 1;
  Money structured_amount = 5;
}

This lifecycle requires coordination but prevents the data corruption that results from abrupt removals.

Handling Enum Evolution

Enums require special care. In proto3, an unknown enum value is preserved as its integer representation and decoded as the zero value by older code. This means the zero value acts as a catch-all for unrecognized variants.

enum PaymentMethod {
  PAYMENT_METHOD_UNSPECIFIED = 0;
  PAYMENT_METHOD_CARD = 1;
  PAYMENT_METHOD_BANK_TRANSFER = 2;
  PAYMENT_METHOD_WALLET = 3;
  // Added in v1.5
  PAYMENT_METHOD_CRYPTO = 4;
}

Adding a new enum value is safe. Old consumers that receive PAYMENT_METHOD_CRYPTO (integer 4) will see PAYMENT_METHOD_UNSPECIFIED (integer 0) because the value is not in their known set. The application logic should handle the unspecified case gracefully, typically by logging a warning and applying a fallback behavior.

Removing an enum value follows the same reserve pattern as fields:

enum PaymentMethod {
  reserved 3;
  reserved "PAYMENT_METHOD_WALLET";
 
  PAYMENT_METHOD_UNSPECIFIED = 0;
  PAYMENT_METHOD_CARD = 1;
  PAYMENT_METHOD_BANK_TRANSFER = 2;
  PAYMENT_METHOD_CRYPTO = 4;
}

Changing the integer assigned to an existing enum name is a breaking change and must never be done in place.

Oneof Evolution and Pitfalls

The oneof construct introduces additional constraints. Adding a new field to a oneof is safe and follows the same rules as adding any other field. However, moving an existing field into or out of a oneof is a breaking change because it alters the wire format and the generated code's accessor patterns.

message Notification {
  string notification_id = 1;
 
  oneof channel {
    EmailNotification email = 10;
    SmsNotification sms = 11;
    PushNotification push = 12;
    // Safe: adding a new variant
    WebhookNotification webhook = 13;
  }
}

A subtle pitfall arises when a oneof field's message type is changed. Even though the field number stays the same, replacing the message type changes the expected wire layout, causing deserialization to fail or produce garbage. Treat the field number and its message type as an inseparable pair.

Versioning Packages for Major Changes

Sometimes a change is so significant that it cannot be made in a backward-compatible way. In these cases, the Nebula registry introduces a new package version:

// nebula/payments/v2/payments.proto
syntax = "proto3";
 
package nebula.payments.v2;
 
message Payment {
  string payment_id = 1;
  Money amount = 2;
  PaymentMethod method = 3;
  repeated PaymentEvent events = 4;
  // v2 introduces a richer status model
  PaymentStatusInfo status_info = 5;
}

The v1 package continues to exist and receive bug fixes. A gateway or adapter service translates between v1 and v2 representations during the migration period. Once all consumers have migrated, the v1 package is frozen and eventually archived.

This approach is expensive: it doubles the maintenance surface during the overlap period. It should be reserved for genuinely incompatible redesigns, not for routine field additions.

Practical Tips for Safe Evolution

Start every schema change by running buf breaking against the current main branch. This catches wire-incompatible changes before they reach code review. In the Nebula CI pipeline, this check is mandatory and cannot be overridden without a designated reviewer's approval.

Write migration runbooks. When a field is deprecated, document the steps each consuming team must take, the monitoring queries that confirm migration progress, and the rollback plan if something goes wrong.

Use feature flags to decouple schema deployment from behavior deployment. A producer can start populating a new field behind a flag, allowing a staged rollout that can be reversed without a code change.

Monitor unknown field counts. If a consumer is consistently encountering unknown fields, it may be falling behind on schema updates. Protobuf libraries in several languages expose hooks for tracking unknown field presence.

Conclusion

Schema evolution is a discipline, not a one-time decision. Protocol Buffers provide a wire format that accommodates change, but only if the team follows the rules: never reuse field numbers, deprecate before removing, reserve what you delete, and version the package when backward compatibility is truly impossible. The Nebula schema registry encodes these rules as automated checks, making safe evolution the path of least resistance for every team that depends on it.