Use case : Data range is very minimum and infrequent between source and target . but still we need in real time . In that case, if I want onboard Kafka , is it possible to make consumer will start consuming the data only when producer is producing message . I knew producer and consumer are decoupled. But what is the best way to handle this use case with Kafka .
1 Answers
0
votes
Kafka Consumer is based on pull architecture. It polls the Kafka broker to fetch new messages (if any).
consumer will start consuming the data only when producer is producing message
If by this you mean - consumer process should come up only after data is produced, that is not provided out of the box in Kafka. You would need to implement logic to start kafka consumer when data is produced.
If it is okay for your consumer to be up all the time, it will continue polling the Kafka broker for new messages in predetermined intervals. Once the producer produces the data, the poll will return the messages that can be processed.
