Kafka: Why broker isn't pull based like consumers

0
votes

I was reading Kafka docs where it was mentioned that:-

  1. Consumers pulls data from broker by requesting from offset.
  2. Producer pushes messages to broker.

Making Kafka consumers pull based make sense that the consumers can drive the pace and broker can store the data for a really long time.

However with producers being push based, How does Kafka make sure that speed mismatch between producer and kafka won't happen? Also producers don't have persistance by design.This seems to be a bigger problem, when producers and brokers are separated over high latency network(internet).

1
apache-kafka
Why a negative vote? I clearly did research in how kafka works, It is not a borad question and it is relevant to programming and a populer tool? – Mangat Rai Modi
I will add more links and details if that would help – Mangat Rai Modi

1 Answers

2
votes

As a distributed commit log, Kafka solves exactly this (impedance mismatch). You produce your events at the rate at which they occur into Kafka, and then you consume them at the rate at which your application can. The data is persisted in Kafka regardless. If your application needs to consume at a greater rate, you scale it out and partition your topic and consume in parallel. Because the data is persisted the only factor is how fast you want to consume the data.