Azure Event Trigger fires multiple times same file

1
votes

I have a blob storage container where I have configured a Event-grid trigger (Blob Created). I am loading this blob storage files through Data factory and many times it will happen that many files may come up in this blob in one shot. May be we can take an example of 20 files.

The good news is my event-grid trigger is firing and the function app is called. However , I can see that sometimes for the same file the event-grid trigger is fired more than once.

Out of these 20 files there are few files which are very large say 300 MB but others are pretty small like in 3KBs. So my doubt is while this 300 MB is fired and it is still processing , parallelly it picks the same 300 MB file again (since it feels that it is still not read) and is saved in db multiple times which is not want I want.

Is Azure Event-grid would be the right approach for this scenario ?

1
azure-functionsazure-eventgrid
it looks like, your subscriber such as a ADF pipeline takes sometime more than 30 seconds, see more details in the docs.microsoft.com/en-us/azure/event-grid/… – Roman Kiss
No... All my files through ADF is copied within around 40 secs. So I believe ADF is not the problem. – amit agarwal
try to read that link again, there is described what is happen during the delivery if the AEG will not received a response from the subscriber (ADF) within the 30 seconds, therefore, when the ADF sent the response latter (such as ~40 seconds), the AEG starts the retrying delivery based on the scheduler time, more details can be found in that link. Also, have a look at the AEG metrics for delivery time, etc. – Roman Kiss
Yeah you are right it takes more than 30s due to which it is retrying. Is there a possibility that if this retrying / response time can be changed to say 10 mins or so or may be it should never trigger if no response is received. – amit agarwal
@RomanKiss, how is this 30 sec period considered , does AEG simply waits for event delivery ACK or it includes event trigger function processing time as well . One more thing , we can't send a HTTP response from azure event trigger function , so this is acknowledgement from event trigger function handled implicitly ? In my scenario , the the event trigger function invoked within 3 sec(T+3sec) after event is published(T) , but the function itself takes around 2/3 mins to complete processing . Still I see the retry happened at (T+3 mins) – Pintu

1 Answers

0
votes

I can see, you opened a new thread for handling your problem which also is based on the Push-Push pattern for the event processing.

I do recommend to use the Push-Pull pattern, where the AEG will deliver an event to the queue storage and then based on the needs can be processed by Azure Queue Trigger function in the concurrent manner, see more details here.

Notes:

  1. The default timeout of the AF for Consumption plan is 5 minutes with a maximum 10 minutes. If this time is not enough for processing a large blob, you should use a Premium Plan where the default timeout is 30 minutes with a maximum time 60 minutes.

  2. In the case when you want to keep your business logic in the ADF pipeline, have a look at here how easy can be invoked the pipeline from the azure function.