Kafka producer difference between flush and poll

Question

We have a Kafka consumer which will read messages and do so stuff and again publish to Kafka topic using below script

producer config :

{
  "bootstrap.servers": "localhost:9092"
}

I haven't configured any other configuration like queue.buffering.max.messages queue.buffering.max.ms batch.num.messages

I am assuming these all will be going to be default values from configuration

queue.buffering.max.messages : 100000
queue.buffering.max.ms : 0
batch.num.messages : 10000

my understanding : When internal queue reaches either of queue.buffering.max.ms or batch.num.messages messages will get published to Kafka in separate thread. in my configuration queue.buffering.max.ms is 0, so every message will be published as soon as when I call produce(). correct me if I am wrong.

My producer snippet:

def send(topic, message):
    p.produce(topic, json.dumps(message), callback=delivery_callback(err, msg))
    p.flush()

from this post i understand that using flush after every message, producer is going to be sync producer . if I use above script it is taking ~ 45ms to publish to Kafka

If I change above snippet to

def send(topic, message):
    p.produce(topic, json.dumps(message), callback=delivery_callback(err, msg))
    p.poll(0)

Is there any performance will be improved ? Can you clarify my understanding.

Thanks

Mickael Maison Mickael Maison · Accepted Answer · 2018-10-01T12:31:46

The difference between flush() and poll() is explained in the client's documentation.

For flush(), it states:

Wait for all messages in the Producer queue to be delivered. This is a convenience method that calls poll() until len() is zero or the optional timeout elapses.

For poll():

Polls the producer for events and calls the corresponding callbacks (if registered).

Calling poll() just after a send() does not make the producer synchronous as it's unlikely the message just sent will already have reached the broker and a delivery report was already sent back to the client.

Instead flush() will block until the previously sent messages have been delivered (or errored), effectively making the producer synchronous.

Kafka producer difference between flush and poll

1 Answers