Kafka consumers logging "Auto offset commit failed" every few minutes

2
votes

I am using the Kafka 0.9 "new" consumer API to connect to Kafka 0.9.0.0. Every couple of minutes, the following error is logged:

2016-01-27T16:38:14,584Z [pool-1-thread-1] ERROR o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed. org.apache.kafka.clients.consumer.internals.SendFailedException: null

Here is a gist of my broker and consumer configs.

There are no corresponding/concerning log messages in the broker logs, and records appear to be flowing through my system just fine. This message is mainly an annoyance, causing distracting alerts and noise.

I'm curious to know what this exception means, but my real questions are:

  1. Should I be concerned by this exception?
  2. If I should not be concerned, is there a way to suppress it by tweaking the consumer/broker configs?
  3. If the answer to the two questions above is "no", what is a good rule for suppressing these messages (I can do it with Logback) so that I don't miss legitimate SendFailedException log messages
1
apache-kafka
I guess auto.commit.interval.ms should be larger than offsets.commit.timeout.ms. but I am not sure. – BAE
@BAE I tried doubling the value of auto.commit.interval.ms, so it was 10000 while leaving offsets.commit.timeout.ms at the default 5000. Still got the error. – maxenglander
I have no idea. maybe github.com/SOHU-Co/kafka-node/issues/122 and codatlas.com/github.com/apache/kafka/trunk/clients/src/main/… help. – BAE

1 Answers

2
votes

I have the same error message in different place (Heron and KafkaSpout)! My finding is the following:

If you use enable.auto.commit = true you must not use consumer.commitAsync();