kafka filtering/Dynamic topic creation

2
votes

I am working on kafka, spark-streaming and trying to achieve Reconciliation.

Producer Message: some 5 fields separated by ',' and one of those would be UUID(unique Identifier) ex: A,B,producer,UUID_1,E

Consumer Message: some 5 fields separated by ',' and one of those would be UUID ex:A,B,consumer,UUID_1,E

Here UUID is the relation between producer and consumer and it is dynamic

In kafka i need some sort of filtering by which all specific producer/consumer messages would be in one particular topic or partition.

I do had an idea as below but unable to implement it:

As per my research dynamic partitions are not allowed, is this correct?

so the only option i see is, Create a dynamic topic based on the received UUID, then route each UUID specific messages to that topic and once the messages got consumed the topic needs to be deleted. Is this the correct approach and if yes how to achieve this?

1
apache-kafka
There is a flag to create a topic dynamically auto.create.topics.enable = true. Hope this helps – Paul Bastide

1 Answers

0
votes

As per my research dynamic partitions are not allowed, is this correct?

Partitions can be increase,but can't be decrease.

Create a dynamic topic based on the received UUID

Dont't do that,it violate the Topic semantic,Topic means same kind messages.
Just set the 'UUID_1' into message 'Key' field,as Kafka send same 'Key' message to the same 'Partition' accroding to the hash of 'Key'.