Apache Flink 1.3.2 connectivity issue with Kafka 1.1.0

0
votes

I'm using a cluster of Apache Flink 1.3.2. We're consuming Kafka messages and since upgrading the broker to 1.1.0 (from 0.10.2) we noticed this error in the log frequently:

 ERROR o.a.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase  - Async Kafka commit failed.
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing offsets.
Caused by: org.apache.kafka.common.errors.DisconnectException: null

Due to this sometimes we experience missing events during processing. We use FlinkKafkaConsumer010 in the job.

Checkpointing is enabled (Interval 10 s, Timeout 1 minute, Minimum pause between checkpoints 5s, Maximum concurrent checkpoints 1. E2E duration on average is under 1s, under half a second even I'd say.) Same settings were used with Kafka 0.10.2 where we don't have this exception.

Update: We have reinstalled Kafka and now we get a warning message but still no events are read

WARN  o.a.flink.streaming.connectors.kafka.internal.Kafka09Fetcher  - Committing offsets to Kafka takes longer than the checkpoint interval. Skipping commit of previous offsets because newer complete checkpoint offsets are available. This does not compromise Flink's checkpoint integrity.
1
apache-flinkflink-streamingflink-cep
Have you enabled checkpointing? If not enabled, Flink commits Kafka offset periodically, and it is not part of checkpoint state. When exception was thrown while consuming messages from Kafka, some message may be missed since probably they have not been processed by all the tasks in streaming job. – BrightFlow
Updated with details, thanks for asking – razvan
Offset commit failure doesn't mean data loss. I'd expect maybe duplicate data, but not data loss. Are you sure you are losing data? If so, there must be something else in the log such as a failure to fetch data due to timeout. DEBUG level logging on the Kafka client might give more insight. – dawsaw
@dawsaw there was, I somehow didn't pasted complete message "ERROR o.a.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Async Kafka commit failed. org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing offsets. Caused by: org.apache.kafka.common.errors.DisconnectException: null" – razvan

1 Answers

1
votes

Turns out this was caused by some connection issues we had in AWS. The framework works well with Kafka 1.1