Azure data factory and Log analytics

Question

I want to perform some validation checks in ADF on my input data and any validation failures want to capture into Azure log analytics.

Can someone guide me how to capture the custom logs into log analytics through Azure Data Factory please.

Any example dataflow/pipeline would be very helpful.

Thanks, Kumar

You can refer to Keeping Azure Data Factory metrics and pipeline-run data — DeepDave-MT

Trent Tamura Trent Tamura · Accepted Answer · 2021-07-12T17:32:37

If I understand correctly you want to be able to get the Azure Monitor logs for ADF and query/store these logs?

Well the good news is most of the information you would want to see is already collected through Azure Monitor.

One of the simple methods to pull the information is to use the Azure Monitor REST API. You can then store the response into a file or table, or you can just query the api for specific pipelines or triggers etc.

Here is a link with example of Authorization and using the Azure Monitor API: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#authenticating-azure-monitor-requests

This is an example of the HTTP URL using the Azure Monitor REST API to get Activity Run data (Dynamic content syntax):

@{concat('https://management.azure.com/subscriptions/', linkedService().SubscriptionID, '/resourceGroups/', linkedService().ResourceGroupName, '/providers/Microsoft.DataFactory/factories/', linkedService().DataFactoryName, '/pipelineruns/', linkedService().RunID, '/queryActivityruns?api-version=2018-06-01')}

Here are all the different ADF Metrics that can be pulled from Azure Monitor: https://docs.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor#data-factory-metrics

Here is an example of a REST Linked service in ADF that can be used to call the rest api:

You could then create a dataset that will pass all the values to the linked service so that you can call the API, and copy the response to a DB, like here:

This particular example is getting the status of a specific Pipeline RunID, but it can be much more broad than that. Here is what the request body looks like so that I could filter to just failed pipelines in the last day:

Azure data factory and Log analytics

1 Answers