Improving performance of Kafka Producer

Question

We're running on apache kafka 0.10.0.x and spring 3.x and cannot use spring kafka as it is supported with spring framework version 4.x.

Therefore, we are using the native Kafka Producer API to produce messages.

Now the concern that i have is the performance of my producer. The thing is i believe a call to producer.send is what really makes the connection to the Kafka broker and then puts the message onto the buffer and then attempts to send and then possibly calls your the provided callback method in the producer.send().

Now the KafkaProducer documentation says that it uses a buffer and another I/O thread to perform the send and that they should be closed appropriately so that there is no leakage of resources.

From what i understand, this means that if i have 100s of messages being sent every time i invoke producer.send() it attempts to connect to the broker which is an expensive I/O operation.

Can you please correct my understanding if i am wrong or maybe suggest a better to use the KafkaProducer?

Answer 1

The two important configuration parameters of kafka producer are 'batch.size' and 'linger.ms'. So you basically have a choice: you can wait until the producer batch is full, or the producer time out.

• batch.size – This is an upper limit of how many messages Kafka Producer will attempt to batch before sending – specified in bytes.

• linger.ms – How long will the producer wait before sending in order to allow more messages to get accumulated in the same batch.

It depends on your use case, but I would suggest to take a closer look on these parameters.

Answer 2

Your understanding is partially right.

As @leshkin pointed out there are configuration parameters to tune how the KafkaProducer will handle buffering of messages to be sent.

However independently from the buffering strategy, the producer will take care of caching established connections to topic-leader brokers.

Indeed you can tune for how long the producer will keep such connection around using the connections.max.idle.ms parameter (defaults to 9 minutes).

So to respond to your original question, the I/O cost of establishing a connection to the broker happens only on the first send invocation and will be amortised over time as long as you have data to send.

Answer 3

In the below conditions you need to configure batch.size, linger.ms & compression.type properties in your kafka prodocer to increase the performance.

1) If records are arriving faster than the kafka producer can send.

2) If you have huge amount of data in the your respective Topic, its really burden to your kafka producer.

3) if you have a bottlenecks

batch.size = 16_384 * 4
 linger.ms 200
compression.type = "snappy"

props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16_384 * 4);
    // Send with little bit buffering
    props.put(ProducerConfig.LINGER_MS_CONFIG, 200);    
  //Use Snappy compression for batch compression.
    props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");

kafka Dzone

Performance tunning

Kafka Perforamnce tunning