Azure alert on Azure Functions "Failed" metric is triggering with no apparent failures

2
votes

I want an Azure Alert to trigger when a certain function app fails. I set it up as a GTE 1 threshold on the [function name] Failed metric thinking that would yield the expected result. However, when it runs daily I am getting notifications that the alert fired but I cannot find anything in the Application Insights to indicate the failure and it appears to be running successfully and completing.

Here is the triggered alert summary:

enter image description here

Here is the invocation monitoring from the portal showing that same function over the past few days with no failures:

enter image description here

And here is an application insights search over that time period showing no exceptions and all successful dependency actions:

enter image description here

The question is - what could be causing a Azure Function Failed metric to be registering non-zero values without any telemetry in Application Insights?

Update - here is the alert configuration

enter image description here

And the specific condition settings-

enter image description here

Failures blade for wider time range:

enter image description here

There are some dependency failures on a blob 404 but I think that is from a different function that explicitly checks for the existence of blobs at paths to know which files to download from an external source. Also the timestamps don't fall in the sample period.

enter image description here

No exceptions:

enter image description here

1
azureazure-functionsazure-application-insightsazure-alerts
Can you share your alert configuration details/screenshot please. – krishg
@KrishnenduGhosh-MSFT Sure! I have updated the description with a screenshot of the alert configuration. Thanks – Gavin H
Thanks. Would be great if you click that condition and share the screenshot. – krishg
Thanks. I am checking. – krishg
@GavinH, not sure what's happening in your side as it's ok at my side. But as a workaround, you can use the custom log search to define the alert which is more trustable. – Ivan Yang

1 Answers

3
votes

Per comment on the question by @ivan-yang I have switched the alerting to use a custom log search instead of the built-in Azure Function metric. At this point that metric seems to be pretty opaque as to what is triggering it and it was triggering every day when I ran the Azure Function with no apparent underlying failure. I plan to avoid this metric now.

My log based alert is using the following query for now to get what I was looking for (an exception happened or a function failed):

requests
| where success == false
| union (exceptions)
| order by timestamp desc

Thanks to @ivan-yang and @krishnendughosh-msft for the help