Difference between FlinkKafkaConsumer and the versioned consumers FlinkKafkaConsumer09/FlinkKafkaConsumer010/FlinkKafkaConsumer011

0
votes

Flink-kafka-consumer has two types of consumers e.g. :

Both of these consumer hierarchies extend same FlinkKafkaConsumerBase class. I wonder what is the design decision behind maintaining the numbered classes? When should we use one or the other?

I noticed that classes with numbered suffix have some more features (E.g. ratelimiting).

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java#L72

EDIT: (following link has stopped working as this file was removed in master branch). https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java#L77

1
apache-kafkaapache-flinkflink-streaming

1 Answers

0
votes

The versioned Kafka consumers (and producers) are built against those versions of the Kafka client, and are intended to each be used with those specific versions of Kafka. The unversioned connectors -- FlinkKafkaConsumer and FlinkKafkaProducer -- are built using the universal client library and are compatible with all versions of Kafka since 0.10.

Note that the versioned consumers and producers for Kafka 0.8 and 0.9 were removed in Flink 1.11, and the versions for 0.10 and 0.11 will be removed in Flink 1.12 (https://issues.apache.org/jira/browse/FLINK-19152).

EDIT:

In some cases it is enough to simply allow the backpressure from the sink to throttle the source. But in other cases (e.g., multiple sources) that may not work well enough.

You will find some discussion about this, and code for doing your own rate limiting, at http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html.