Data Factory - Data Lake File Created Event Trigger fires twice

Question

I'm developing Pipeline in Azure Data Factory V2. It has very simple Copy activity. The Pipeline has to start when a file is added to Azure Data Lake Store Gen 2. In order to do that I have created a Event Trigger attached to ADLS_gen2 on Blob created. Then assigned trigger to pipeline and associate trigger data @triggerBody().fileName to pipeline parameter.

To test this I'm using Azure Storage Explorer and upload file to data lake. The problem is that the trigger in Data Factory is fired twice, resulting pipeline to be started twice. First pipeline run finish as expected and second one stays in processing.

Has anyone faced this issue? I have tried to delete the trigger in DF and create new one but the result was the same with new trigger. enter image description here

enter image description here

dtape dtape · Accepted Answer · 2019-11-07T02:12:37

I'm having the same issue myself.

When writing a file to ADLS v2 there is an initial a CreateFile operation and a FlushWithClose operation and they are both triggering a Microsoft.Storage.BlobCreated event type.

https://docs.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage

If you want to ensure that the Microsoft.Storage.BlobCreated event is triggered only when a Block Blob is completely committed, filter the event for the FlushWithClose REST API call. This API call triggers the Microsoft.Storage.BlobCreated event only after data is fully committed to a Block Blob.

https://docs.microsoft.com/en-us/azure/event-grid/how-to-filter-events

You can filter out the CreateFile operation by navigating to Event Subscriptions in the Azure portal and choosing the correct topic type (Storage Accounts) and subscription and location. Once you've done that you should be able to see the trigger and update the filter settings on it. I removed CreateFile.

Data Factory - Data Lake File Created Event Trigger fires twice

2 Answers