Kafka consumer fails silently to connect to broker

1
votes

We had an issue today where a Kakfa consumer failed to connect to any brokers, and there was no error message in the logs. It failed to connect as it was connecting over SSL and was pointing to the wrong truststore. But although in this instance it was due to an SSL failure, I have seen other times where the consumer fails to connect, and there be no error message in the logs.

Is there a way to configure a Kafka consumer so that an error message is logged if the consumer cannot connect to a broker? I am looking for a generic solution, not specific to SSL connection issues.

(We are using: org.apache.kafka:kafka-clients:0.9.0.1 and org.springframework.kafka:spring-kafka:1.0.4.RELEASE)

1
apache-kafkaspring-kafka
Have you set up log4j? – Giorgos Myrianthous
spring-kafka:1.0.4.RELEASE - that is really old; all 1.x. user are recommended to upgrade to at least 1.3.7 - KIP-62 allows a much simpler threading model. I have seen this before (mismatched SSL), this client just hangs with no logs. The latest spring-kafka is 2.2.0 (which uses the kafka 2.0.0 client, which has timeouts which might help with dealing with problems like this). – Gary Russell
@GiorgosMyrianthous We are using SLF4J with Logback and I can see INFO level log messages from org.apache.kafka... in the logs. – vegemite4me
@vegemite4me Can you turn your logs into DEBUG then? – Giorgos Myrianthous
@GiorgosMyrianthous Yes I could. But it does not seem right to have to increase logging to DEBUG (even for a single class) for a situation like this which happened when running in production. – vegemite4me

1 Answers

2
votes
  • consider upgrading your client to newer version of kafka-clients (in later versions some operations like seek started to throw instead of hanging - that would let you know of issues faster)
  • set up log4j, especially org.apache.kafka.clients.NetworkClient class (it logs if it cannot find a broker)
  • consider configuring default.api.timeout.ms or in older versions metadata.fetch.timeout.ms to have timeouts faster