Many of the systems I have worked on in recent years, including the platform I lead at Nike, have included Kafka and Confluent Schema Registry. The Confluent Schema Registry is a vital piece of the Kafka ecosystem. It is used to persist the various schemas and their versions of the events that are published and consumed via Kafka.

When working on complex systems, I like to try and isolate the various parts and learn them separately, rather than as part of the whole. I think this allows us to see their APIs and functionality better. It may also open up some new possibilities we may miss when viewing it as a piece of the whole.

I’ve begun working on a new system that is adding Kafka and Schema Registry, to jog my memory and check out new functionality, I’ve created a GitHub repo that contains a docker-compose file for running the service locally. I’ve also included a README full of simple example commands that can be used to learn the API.

Beyond just registering schemas, there are some interesting topics to explore:

Schemas can be defined in various formats:
- JSON
- AVRO
- Protobuf

Schemas have versions and enforce compatibility modes for evolutions.

Compatibility modes:
- *Backward (recommended)* — receiver can read both current and previous versions.
- *Backward All* — receiver can read current and all previous versions.
- *Forward* — sender can write both current and previous versions.
- *Forward All* — sender can write current and all previous versions.
- *Full* — combination of Backward and Forward.
- *Full All* — combination of Backward All and Forward All.
- *None* — no compatibility checks performed.
- *Disabled* — prevent any versioning for this schema.

If you are new to Kafka, Confluent has some excellent docs. You can also download the Definitive Guide book if you are willing to offer up some personal info.

Music companion of this post:
Floating Ashes - Impulse

What am I working on currently:
I am working on several things at the moment, an API for my startup–Schemabook, and getting onboarded onto a Rust based system that uses WASM on the frontend.