Kafka producer TimeoutException: Expiring 1 record(s)

Question

I am using Kafka with Spring-boot:

Kafka Producer class:

@Service
public class MyKafkaProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    private static Logger LOGGER = LoggerFactory.getLogger(NotificationDispatcherSender.class);

    // Send Message
    public void sendMessage(String topicName, String message) throws Exception {
        LOGGER.debug("========topic Name===== " + topicName + "=========message=======" + message);
        ListenableFuture<SendResult<String, String>> result = kafkaTemplate.send(topicName, message);
        result.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
            @Override
            public void onSuccess(SendResult<String, String> result) {
                LOGGER.debug("sent message='{}' with offset={}", message, result.getRecordMetadata().offset());
            }

            @Override
            public void onFailure(Throwable ex) {
                LOGGER.error(Constants.PRODUCER_MESSAGE_EXCEPTION.getValue() + " : " + ex.getMessage());
            }
        });
    }
}

Kafka-configuration:

spring.kafka.producer.retries=0
spring.kafka.producer.batch-size=100000
spring.kafka.producer.request.timeout.ms=30000
spring.kafka.producer.linger.ms=10
spring.kafka.producer.acks=0
spring.kafka.producer.buffer-memory=33554432
spring.kafka.producer.max.block.ms=5000
spring.kafka.bootstrap-servers=192.168.1.161:9092,192.168.1.162:9093

Let's say I have sent 10 times 1000 messages in topic my-test-topic.

8 out of 10 times I am successfully getting all messages in my consumer but sometimes I am getting this below error:

2017-10-05 07:24:11, [ERROR] [my-service - LoggingProducerListener - onError:76] Exception thrown when sending a message with key='null' and payload='{"deviceType":"X","deviceKeys":[{"apiKey":"X-X-o"}],"devices...' to topic my-test-topic

and org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-test-topic-4 due to 30024 ms has passed since batch creation plus linger time

Is this error you are describing from the producer or the consumer? — adarshr
So, your batch is too slow for such a "low" request.timeout.ms. Try to make batch-size a bit lower — Artem Bilan
Isn't 30 second enough?( I am new to Kafka, please bear with me) — Prakash Pandey
I don't know, but according your error you are really exceeding those 30 secs: due to 30024 ms has passed — Artem Bilan

michalbrz michalbrz · Accepted Answer · 2017-12-30T12:49:08

There are 3 possibilities:

Increase request.timeout.ms - this is the time that Kafka will wait for whole batch to be ready in buffer. So in your case if there are less than 100 000 messages in buffer, timeout will occur. More info here: https://stackoverflow.com/a/34794261/2707179
Decrease batch-size - related to previous point, it will send batches more often but they will include fewer messages.
Depending on message size, maybe your network cannot catch up with high load? Check if your throughput is not a bottleneck.

Kafka producer TimeoutException: Expiring 1 record(s)

3 Answers