Apache Kafka with centralized Avro schema

Question

We use Apache Kafka( not confluent kafka ) 0.10. We would like to setup AVRO schema with kafka. I have avro schema as follows.

{
  "namespace": "Rule",
  "type": "record",
  "name": "RuleMessage",
  "fields": [
    {
      "name": "station",
      "type": "string"
    },
    {
      "name": "model",
      "type": "string"
    }
}

Serializing message like,

public byte[] serializeMessage(EventMessage eventMessage) throws IOException {

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
        DatumWriter<EventMessage> writer = new SpecificDatumWriter<EventMessage>(EventMessage.getClassSchema());
        writer.write(eventMessage, encoder);
        encoder.flush();
        out.close();
        return out.toByteArray();
    }

This is working as expected.

But, would like to setup an Avro Schema at the topic level, so that the topic will reject messages if the message is not meeting the avro schema.

Is there anyway, I could do this with Apache Kafka 0.10.

Thanks

Gwen Shapira Gwen Shapira · Accepted Answer · 2016-11-01T14:19:11

You can use Confluent's Schema Registry (its open source and Apache licensed) with Apache Kafka 0.10.0 to associate a schema with a topic. It arrives with Avro Serializers/DeSerializers that automatically validate the Avro schemas in exactly the way you requested.

Please note that the is no such thing as "Confluent Kafka" - it would be a trademark violation to have it. Confluent simply packages Apache Kafka in its distribution for convenience, but since the Schema Registry is on github, you can use that without using Confluent packaging if you prefer.

Apache Kafka with centralized Avro schema

1 Answers