Kafka broker 1.10, clients using API 0.10.2

Question

Should we update our Scala Kafka client library dependency (currently 0.10.2) to match the Kafka version of the broker (v1.1.0) ?

The Kafka 0.10.2 Documentation mentions

Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.10.2 clients can talk to version 0.10.0 or newer brokers

Are there any adverse effects when the client API version lags behind the server version? More importantly, can we safely update our Kafka client API library from 0.10.2 to 1.10?

Answer 1

While the brokers are now compatible with older clients, there are a few drawbacks in using older clients.

The main one is Message conversion. Between 1.1 and 0.10.2, the record format has changed. So, by default, older clients will force brokers to convert messages when producing and consuming. Conversion is pretty memory intensive and has a performance cost. See http://kafka.apache.org/documentation/#upgrade_11_message_format

Then obviously old clients are unable to use new features. Between 0.10.2 and 1.1, there's a ton a nice features like Exactly Once semantics, better authentication feedback on failure, Admin operations, etc