We run an hybrid application that runs on different Azure Roles (2 Web Roles + 2 Worker Roles). Last weekend something went wrong and the service went into "Unresponsive" state and stayed into that state for two days (!) without being rebooted.
We decided than to integrate Azure Application Insights because we cannot stand a 2 days down without even know.
What I'd like to have is kinda of a heartbeat of my application. One of our worker roles has different services running concurrently. I'd like to monitor if this services are running and what are their performance (on a metric defined by me, let say "number of messages processed in a minute").
I would like to receive an alert if this metric, let say, goes down (or up) a threshold. I tried with a small demo application but I couldn't do that.
What I did, with Azure Application Insights API on my C# demo app: 1. inside an infinite loop with a 10 seconds wait after each loop, tracked a StartOperation 2. inside this StartOperation, tracked a TrackMetric passing a random value from 0 to 10 3. checked if everything was working on Azure (and it was) 4. defined an alert saying that an email has to be sent if that metric was less or equal to 1 in five minutes
Nothing arrived, but everything was correctly running. Than I stopped my service, I saw events dropping in Azure, but no alert raised. Is that normal?
How do you check a case like mine?
Thanks Marco