Kafka - Deserializing the object in Consumer

Question

We are considering to use Kafka in our for messaging and our applications are developed using Spring. So, we have planned to use spring-kafka.

The producer puts the message as HashMap object into the queue. We have JSON serializer and we assumed that the map will be serialized and put into the queue. And here is the producer config.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
        key-serializer: org.springframework.kafka.support.serializer.JsonSerializer
        value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

On the other hand, we have a listener which listens to the same topic where the producer has published the message. Here is the consumer config:

spring:
   kafka:
       consumer:
            group-id: xyz
            key-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
            value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer

Our listener method:

  public void listener(SomeClass abx)

We were expecting the json will be de-serialized and an object of type "SomeClass" will be generated. But apparently, it throws de-serialization exception.

We saw few articles and the suggestion was to do something like:

 @Bean
  public ConsumerFactory<String, Car> consumerFactory() {
    return new DefaultKafkaConsumerFactory<>(consumerConfigs(), new StringDeserializer(),
        new JsonDeserializer<>(Car.class));
  }

We don't want to write some code for creating the Deserializer. Is there any boilerplate thing which we are missing? Any help will be appreciated!!

Docs for serialization/deserialization: docs.spring.io/spring-kafka/reference/htmlsingle/#serdes Looks like it should work just with @KafkaListener annotation — Pospolita Nikita
They are suggesting to do something like new JsonDeserializer<>(Bar.class); which I dont want todo, as it will grow when we have more listeners with different types of object — Thiru
I don't think it's a problem to have a single configuration class for Kafka. it's just a part of app configuration. But I will watch this issue, cause it's interesting if the solution exists. — Pospolita Nikita
You can set the required type using a property - see my answer. — Gary Russell

Gary Russell Gary Russell · Accepted Answer · 2018-06-29T12:03:06

See the boot documentation. In particular:

You can also configure the Spring Kafka JsonDeserializer as follows:

spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

spring.kafka.consumer.properties.spring.json.value.default.type=com.example.Invoice

spring.kafka.consumer.properties.spring.json.trusted.packages=com.example,org.acme

Kafka - Deserializing the object in Consumer

1 Answers