Azure Data Factory: Configure Frequency

Question

I am planning to use Azure Data Factory as a trigger for data lake analytics jobs in a project. The data lake jobs will calculate key figures based on sensor input data that is processed by StreamAnalytics and stored in Data Lake. These jobs should calculate the values every ~5 minutes.

According to Microsoft documentation it is not possible to configure intervals / frequencies smaller than 15 minutes.

Anybody faced the same problem and found a solution or is it better to use a different tool as Azure Data Factory in this scenario?

Answer 1

As you already noticed - the minimal configurable interval is 15 minutes. If you look for tinier intervals you should look at streaming solutions amd not Data Factory. Because there is nit real context in your question, I cannot suggest you which service you should look at. But Azure Logic apps may be a good candidate as there you can have down to 1 interval.

Answer 2

In ADF one cant have a frequency lesser than 15 mins advisable. https://docs.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution

For your use case you can take a look at Azure Stream Analytics which is meant for streaming ingestion from IOT https://docs.microsoft.com/en-us/azure/stream-analytics/

Answer 3

You could do this with multiple copies of your pipeline with different names and different availability configurations in your output datasets. For example, 3 pipelines with 3 datasets set to these 3 availability configs would cover each 5-min interval:

"availability":    
{    
    "frequency": "Minute",        
    "interval": 15,    
    "anchorDateTime":"2017-10-01T00:00:00"    
}
"availability":    
{    
    "frequency": "Minute",        
    "interval": 15,    
    "anchorDateTime":"2017-10-01T00:00:05"    
}
"availability":    
{    
    "frequency": "Minute",        
    "interval": 15,    
    "anchorDateTime":"2017-10-01T00:00:10"    
}

Note you might need to implement some kind of synchronization lock if you don't want the executions to overlap.

Answer 4

I am using ADF v2 and I can have a frequency lesser than 15 mins. for example you can see that I have a trigger for each minute.