How to perform Event based data ingestion using Azure Data Lake Storage Gen2 and Azure Data factory V2?

Question

Recently we came across a scenario where our source and sink location are of ADLS Gen2 type. Now we got one interesting use case wherein we have to push data from source to sink with the help of ADF V2. Having said that, its not just normal copy activity we are expecting but we need to perform this activity on an event basis.

While going through the ADLS Gen2 documents found that ADLS Gen2 yet to support "Azure Event Grids" and that's the reason though we are able to configure ADF's event-based triggers they did not work.

Can anyone suggest me to tackle this situation, since Azure Event Gird is not supported at this instance of time we don't believe we can achieve this with Azure Event Hubs and their integration with ADF?

Thanks.

Answer 1

From my repro, currently event based trigger are supported only on v2 storage accounts.

Data Factory is now integrated with Azure Event Grid, which lets you trigger pipelines on an event.

Note: This integration supports only version 2 Storage accounts (General purpose).

Azure Event Grid doesn't receive events from Azure Data Lake Gen2 accounts because those accounts don't yet generate them.

For more details, refer “Known issues with Azure Data Lake Storage Gen2”.