Application Insights custom metrics with scale-out / multiple instances: how do I combine or disambiguate across instances?

Question

I have an Azure app service web app. I am tracking some custom metrics from this web app, using TelemetryClient.TrackMetric. I now want to scale out my app so that I have multiple instances of the app running (initially I will scale to 2 instances). Once I do this, each instance will start sending values for my custom metrics into the Application Insights data store.

When I look at my AI data inside the Azure portal Metrics Explorer or "Analytics" view, I believe I will see one metric, with the data from both my instances bundled together and averaged (probably I will see alternating datapoints, from instance 1 then 2 then 1 then 2 and so on). This is not very helpful to me. Ideally I want to be able to see two datasets, one for each of my two instances. Also ideally I would like to see a dataset representing the sum of the metric values from the two instances.

Does the API / framework provide any features to help me do this? What would be a good approach?

Answer 1

In my opinion, You should have the default collected property "Cloud role instance" available on the custom metric to segment/ filter on. Can you give it a try and see if it shows you the relevant information per instance?

Answer 2

According to the answers of this question Windows Azure Websites - Get Instance Identifier there is a way to distinguish one instance of the other.

Using that information you can include that identifier in the custom properties of your metric like this:

var metric = new MetricTelemetry("MyMetric", 8);
metric.Properties.Add("InstanceIdentifier", instanceIdentifier);
telemetry.TrackMetric(metric);

Now, not only can you create custom charts in the Dashboard, but also using Application Insights Analytics (https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics) you can query the metrics per instance or combined and create tables, charts etc. based on the results.

For example:

customMetrics
| extend instanceIdentifier = tostring(customDimensions.InstanceIdentifier)
| summarize sum(value), avg(value)  by instanceIdentifier, name
| project instanceIdentifier, name , avg_value, sum_value