Create multiple Event hub receiver to process huge volume of data concurrently

1
votes

Hi I have a event hub with two consumer group.

Many device are sending data to my event hub any I want to save all message to my data base.

Now data are getting send by multiple device so data ingress is to high so in order two process those message i have written one EventHub Trigger webjob to process the message and save to database.

But since saving these message to my data base is time consuming task or I can say that receiver speed is slow then sender speed.

So is there any way two process these message faster by creating multiple receiver kind of thing.

I have create two event receiver with different consumer group but I found that same message is getting processed by both trigger function so now duplicate data are getting save in my data base.

So please help me to know how I can create multiple receiver which will process unique message parallel.

Please guys help me...

1
azureazure-webjobsazure-eventhub
Possible duplicate of How to configure Receiver batch size for Azure Functions EventHub listener? – Kannaiyan
Are you handling messages in batches? Can you process messages from a batch in parallel? – Mikhail Shilkov

1 Answers

0
votes

Creating multiple consumer groups won't help you out as you found out yourself. Different consumer groups all read the same data, but they can have their own speed.

In order to increase the speed of processing there are just 2 options:

  1. Make the process/code itself faster, so try to optimize the code that is saving the data to the database
  2. Increase the amount of partitions so more consumers can read the data from a given partition in parallel. This means however that you will have to recreate the Event Hub as you cannot increase/decrease the partition count after the Event Hub is created. See the docs for guidance.

about 2.: The number of concurrent data consumers is equal to the number of partitions created. For example, if you have 4 partitions you can have up to 4 concurrent data readers processing the data.

I do not know your situation but if you have certain peaks in which the processing is too slow but it catches up during more quiet hours you might be able to live with the current situation. If not, you have to do something like I outlined.