Data factory Diagnostic settings

Question

We have a requirement wherein we need to periodically analyze pipeline executions for the last quarter to identify job failures and execution trends for reporting needs

The operations team already has azure log analytics created for other azure offerings, in case if we want to leverage azure Log Analytics to query the data and create reporting charts what diagnostic configuration should i need to enable?

enter image description here

We are thinking of leveraging log analytics but Data Factory stores pipeline-run data for only 45 days, so should we use log analytics or should we map the data to storage account (to maintain historical trends of before 45 days)

Answer 1

you can use diagnostic setting to both Azure Storage and Log Analytics. So the final goal is to have reporting and query via Log analytics, it would be preferred to use Log Analytics as data that is collected via Log Analytics will remain in the AzureDiagnostics table until it's removed according to the retention setting for the workspace and is not governed by the resource :)