Skip to content

Software Quality

Schema compatibility checking

Refusing a schema change that would break the systems reading data written under the previous version.

Schema compatibility checking compares a new version of a shared schema with the version before it, and rejects the change when a reader on one version could no longer read data written under the other. The schema is the contract between a producer and consumers it may not know: the messages on an event topic, the operations of an HTTP API, the columns of a table other teams query. Producer and consumers deploy on their own schedules, so for a while old readers meet data written under the new schema, and new readers meet data written under the old one. The check decides compatibility from the two schemas alone, for every record they admit, before any such data exists.

Compatibility rules

A rule names which side may upgrade first:

  • Backward — consumers on the new schema can read data written with the previous one, so consumers upgrade first. It is the default in Confluent Schema Registry.
  • Forward — consumers on the previous schema can read data written with the new one, so producers upgrade first.
  • Full — both directions.

Each rule also has a transitive form, which checks the new version against every earlier version rather than only the latest. Compatibility does not chain: three versions can each be compatible with their predecessor while the third cannot read data written under the first. A topic that keeps months of events, or a consumer several versions behind, needs the transitive form.

What counts as breaking depends on the format. Protobuf identifies a field on the wire by its number, not its name, so renaming a field breaks the code generated from the schema but not the encoded bytes; buf's rule sets therefore range from FILE, which protects generated code, down to WIRE, which protects only the binary encoding.

What it catches

  • Removed fields and columns. A consumer that reads a field the producer stopped writing fails, or reads nulls where it expects values.
  • Changed types. A column changed from integer to text, or a field from a number to a string, breaks the readers that decode the old type.
  • New requirements on input. An API parameter or a message field made required rejects requests and messages from clients that do not send it.
  • Removed operations. Deleting an HTTP endpoint fails every client that still calls it.
  • Breaking changes shipped as minor releases. The check runs on every change, so it does not depend on the producer judging the change correctly or choosing the right version number.

What compatibility checking does not catch:

  • a change in meaning that keeps the shape, such as a price that switches from cents to euros or a status value reused for a new state;
  • data written without going through the check: a producer that bypasses the registry, or a subject whose rule is set to none;
  • a change in behavior behind an unchanged schema.

Where the check runs

A schema registry checks each new version when a producer registers it, and refuses a version that fails the subject's rule, so producers that go through the registry cannot write data under it. A CI check compares the schema on a branch with the version on the main branch or the deployed one, and fails the pull request. The CI check reaches the author while an additive change is still an option; the registry check also covers producers whose schemas never pass through that pipeline.

Tools

  • buf — buf breaking compares a Protobuf schema with a past version from a Git repository, a Buf Schema Registry module or a tarball, under rule sets from FILE (generated code) to WIRE (binary encoding).
  • Apicurio Registry — open-source schema registry that applies a backward, forward or full rule, transitive or not, to each new Avro, Protobuf or JSON Schema version. Confluent Schema Registry and AWS Glue Schema Registry apply the same rules.
  • oasdiff — compares two OpenAPI descriptions; its breaking command lists only the changes that break existing clients, and it runs in CI through a GitHub Action.
  • dbt — model contracts: with a contract enforced, dbt compares the project with its previous state and raises an error when a column is removed, a column's type changes, or a constraint is removed or changed.
  • Data Contract CLI — datacontract breaking fails when a change to a data contract breaks a guarantee its consumers rely on.

When to use, when not

Use:

  • On every schema whose consumers deploy separately from its producer: event topics, internal and public HTTP APIs, warehouse tables other teams query.
  • As a CI gate on the pull request that changes the schema, where the author can still turn a removal into an addition.
  • With a transitive rule where old data stays readable, such as a topic that retains events for replay.

Don't:

  • On a schema read only by code released in the same deploy as its producer. No window exists in which the two sides run different versions.
  • As the only check on a change in meaning. A shape check passes a field whose values changed units; consumer-driven contract tests replay what consumers actually expect.
  • By setting a subject's rule to none to push one breaking change through. The registry then accepts any later version as well, until someone restores the rule.

Evidence

The support for the check itself is definitional, not statistical: a consumer on a schema that passes a backward rule can read every record written under the previous version, for the change kinds the tool's rules model.

  • Producers ship breaking changes without signaling them. Of 3,075 OpenAPI histories mined from GitHub, 2,487 APIs introduced breaking changes, and in the best case only 517 of them consistently released those changes as major versions (Serbout and Pautasso 2024)1.
  • Breaking changes slip past manual assessment and break consumers. In interviews with 17 practitioners at 11 companies evolving HTTP and message-broker APIs, 9 said developers sometimes overlooked that a change was breaking and published it without versioning or notice, and 10 reported that their system or a consuming one had broken because the other side did not correctly assess or announce a breaking change (Lercher et al. 2024)2.

Classification

  • Quality dimensions: Functionality: compatibility (a consumer built against the earlier schema can still read what the producer writes), Reliability: reliability (a breaking change is refused before it ships, rather than failing in a consumer in production).
  • Area: Schemas that systems deployed separately depend on: event and message schemas (Avro, Protobuf, JSON Schema) held in a schema registry, HTTP API descriptions (OpenAPI), and the published columns of warehouse tables (dbt model contracts, data contracts).
  • Guarantee: Exhaustive within the tool's rules — the verdict covers every record the two schemas admit, not a sample of them; a change the rules do not model passes.

Referenced by

References

  1. Serbout, Souhaila, and Cesare Pautasso. 2024. "How Many Web APIs Evolve Following Semantic Versioning?" Web Engineering (ICWE 2024), Lecture notes in computer science. https://doi.org/10.1007/978-3-031-62362-2_25. ↩

  2. Lercher, Alexander, Johann Glock, Christian Macho, and Martin Pinzger. 2024. "Microservice API Evolution in Practice: A Study on Strategies and Challenges." Journal of Systems and Software 215. https://doi.org/10.1016/j.jss.2024.112110. ↩