Created
December 19, 2025 16:05
-
-
Save jbreckmckye/45030f697b0a627c76b78c063673fdd1 to your computer and use it in GitHub Desktop.
Kafka schema registration and reconcilliation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ,----------. ,-----------------. | |
| | Producer |------------------------>| Schema Registry | | |
| `----------' `-----------------' | |
| Hi. Can I register a schema S1? | |
| It looks like this {avsc} | |
| ,----------. ,-----------------. | |
| | Producer |<------------------------| Schema Registry | | |
| `----------' `-----------------' | |
| Sure. This looks | |
| compatible with what | |
| we had before. | |
| Let's use ID 1234. | |
| ,----------. ,---------. | |
| | Producer |------------------------>| Kafka | | |
| `----------' `---------' | |
| I am producing a message | |
| ┌─────┬────────────┐ | |
| |Byte | Content | | |
| ├─────┼────────────┤ | |
| |1 | Magic byte | | |
| ├─────┼────────────┤ | |
| |2..5 | Schema ID | "(= 1234)" | |
| ├─────┼────────────┤ | |
| |6+ | Message | | |
| └─────┴────────────┘ | |
| ...ok | |
| LATER... | |
| ,----------. ,---------. | |
| | Consumer |------------------------>| Kafka | | |
| `----------' `---------' | |
| Do you have a | |
| message for me? | |
| yes! | |
| it is | |
| ┌─────┬────────────┐ | |
| |Byte | Content | | |
| ├─────┼────────────┤ | |
| |1 | Magic byte | | |
| ├─────┼────────────┤ | |
| |2..5 | "1234" | | |
| ├─────┼────────────┤ | |
| |6+ | Message | | |
| └─────┴────────────┘ | |
| ,----------. ,-----------------. | |
| | Consumer |------------------------>| Schema Registry | | |
| `----------' `-----------------' | |
| I've received a message | |
| with schema ID 1234. | |
| Can you tell me what that is? | |
| What is this WRITER SCHEMA? | |
| ,----------. ,-----------------. | |
| | Producer |<------------------------| Schema Registry | | |
| `----------' `-----------------' | |
| Yes. I recognise that | |
| ID. It is this AVSC. | |
| Here you go. | |
| ,----------. | |
| | Consumer | | |
| `----------' | |
| "Hmm. I have a READ SCHEMA in my own memory." | |
| But I don't recognise this WRITER SCHEMA. | |
| ,----------. ,-----------------. | |
| | Consumer |------------------------>| AVSC Library | | |
| `----------' `-----------------' | |
| Can you tell me how to | |
| RECONCILE these two schemas? | |
| Yes. As it meets the | |
| compatibilty requirement, | |
| we can read it like so. | |
| Skip these fields. Use | |
| these default values. | |
| ,----------. | |
| | Consumer | | |
| `----------' | |
| Alright, now I know how to process this message! |
Author
jbreckmckye
commented
Dec 19, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment