Skip to content

Instantly share code, notes, and snippets.

@jbreckmckye
Created December 19, 2025 16:05
Show Gist options
  • Select an option

  • Save jbreckmckye/45030f697b0a627c76b78c063673fdd1 to your computer and use it in GitHub Desktop.

Select an option

Save jbreckmckye/45030f697b0a627c76b78c063673fdd1 to your computer and use it in GitHub Desktop.
Kafka schema registration and reconcilliation
,----------. ,-----------------.
| Producer |------------------------>| Schema Registry |
`----------' `-----------------'
Hi. Can I register a schema S1?
It looks like this {avsc}
,----------. ,-----------------.
| Producer |<------------------------| Schema Registry |
`----------' `-----------------'
Sure. This looks
compatible with what
we had before.
Let's use ID 1234.
,----------. ,---------.
| Producer |------------------------>| Kafka |
`----------' `---------'
I am producing a message
┌─────┬────────────┐
|Byte | Content |
├─────┼────────────┤
|1 | Magic byte |
├─────┼────────────┤
|2..5 | Schema ID | "(= 1234)"
├─────┼────────────┤
|6+ | Message |
└─────┴────────────┘
...ok
LATER...
,----------. ,---------.
| Consumer |------------------------>| Kafka |
`----------' `---------'
Do you have a
message for me?
yes!
it is
┌─────┬────────────┐
|Byte | Content |
├─────┼────────────┤
|1 | Magic byte |
├─────┼────────────┤
|2..5 | "1234" |
├─────┼────────────┤
|6+ | Message |
└─────┴────────────┘
,----------. ,-----------------.
| Consumer |------------------------>| Schema Registry |
`----------' `-----------------'
I've received a message
with schema ID 1234.
Can you tell me what that is?
What is this WRITER SCHEMA?
,----------. ,-----------------.
| Producer |<------------------------| Schema Registry |
`----------' `-----------------'
Yes. I recognise that
ID. It is this AVSC.
Here you go.
,----------.
| Consumer |
`----------'
"Hmm. I have a READ SCHEMA in my own memory."
But I don't recognise this WRITER SCHEMA.
,----------. ,-----------------.
| Consumer |------------------------>| AVSC Library |
`----------' `-----------------'
Can you tell me how to
RECONCILE these two schemas?
Yes. As it meets the
compatibilty requirement,
we can read it like so.
Skip these fields. Use
these default values.
,----------.
| Consumer |
`----------'
Alright, now I know how to process this message!
@jbreckmckye
Copy link
Author

image (4)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment