Publish Kafka objects in batches

0
votes

I am using KafkaProducer.send() method to publish records to Kafka. This is an asynchronous method. My application is publishing about 20k records to Kafka. A lot of times after successfully sending about 10k records I get Timeout Exception. I increased batch.size and linger.ms which reduced the problem but i still get Timeout Exception sometimes. Is there a way to send only 10k records at a time wait for them to complete and then send the next batch?? I do not want to make it synchronous by using send.get() because that will make it very slow. This is the exception I am getting.

Error publishing object on partition 0 org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)
1
javaapache-kafkakafka-producer-api
Did you try to use flush()? This will block your producer and make all the messages "send ready". – Manoj Vadehra
Here's another question with exactly same exception: stackoverflow.com/questions/46649748/… – Manoj Vadehra
I am using flush() after sending every message. I do not think that is working for me. – Anuja Barve
why are you using flush() after each record? – Deadpool

1 Answers

0
votes

Did you try set retries=3 to the producer config?
And probably in case of record expiration you should not increase batch.size but decrease it.