Azure Application Insights - Not recording all requests on high traffic situations

Question

It seems "Azure Application Insights" not recording all the requests on high traffic environment.

For example when we testing the .Net Core2.1 Web API app which is deployed on "Azure Service Fabric" with 10,000 requests for the period of 30 mins, can able to retrieve details of all the requests from "Azure Application Insights" via KQL using datetime stamp as filter, no problem.

When we increase the load to 100,000 requests for the period of 30 mins, only around 5-10% of the requests get recorded on "Azure Application Insights".

Why "Azure Application Insights" misses ingest/recording on high traffic environment which serves approximately 60 requests/second ?

Any extra configuration required? (or) Below one line code to ingest details of requests served by Azure Service Fabric service, is not enough? please clarify

SDK used,

Code used on Azure Service Fabric for ingestion

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                            .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext))) // Azure Service Fabric Telemetry Initializer
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                   .UseApplicationInsights()
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

Sample query with item count

Sampling not enabled from Azure portal

It might be due to sampling, see docs.microsoft.com/en-us/azure/azure-monitor/app/sampling — Peter Bons
But I haven't mentioned/configured sampling in my application. All I have done is included the one & only SDK (Microsoft.ApplicationInsights.ServiceFabric.Native v2.1.1), then one line of code(.AddSingleton((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext)). Here where comes sampling? is sampling default behavior? Also sampling didn't come in to effect when I make 10,000 requests for the period of 30 mins.Only when I increase count to 100,000 requests for the period of 30 mins then record missing happening on Azure Application Insights — 191180rk
From the docs: Adaptive sampling is enabled by default in all the latest versions of the Application Insights ASP.NET and ASP.NET Core Software Development Kits (SDKs) . Same applies to service fabric I think. On the page you can find info on how to detect if sampling is in use and how to disable/modify it. — Peter Bons
10.000 requests in 30 sec. is about 5 rps. that probably falls within the range before sampling kicks in. — Peter Bons
To check whether sampling is active - please check "itemCount" property in Kusto. If it is more than 1 then it means data is being sampled. — ZakiMa

krishg krishg · Accepted Answer · 2020-11-02T07:40:31

You can disable adaptive sampling from code. Note instead of using deprecated .UseApplicationInsights() (removed that below), I am using .AddApplicationInsightsTelemetry below.

  return new WebHostBuilder().UseHttpSys()
                                .ConfigureServices(
                                    services => services
                                        .AddSingleton<StatelessServiceContext>(serviceContext)
                                        .AddSingleton<ServiceFabricAppContext>(new ServiceFabricAppContext(){
                                                NodeName = serviceContext.NodeContext.NodeName,
                                                ServiceHostIP=serviceContext.NodeContext.IPAddressOrFQDN,
                                                ServiceHostPort=FabricRuntime.GetActivationContext().GetEndpoints()[0].Port
                                        } )
                                       .AddSingleton<ITelemetryInitializer>((serviceProvider) => FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(serviceContext)) // Azure Service Fabric Telemetry Initializer
                                       .AddApplicationInsightsTelemetry(o => 
                                            {
                                                o.EnableAdaptiveSampling = false; // disabling adaptive sampling
                                            }))
                                .UseContentRoot(Directory.GetCurrentDirectory())
                                .UseStartup<Startup>()
                                .UseEnvironment(environment)
                                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                .UseUrls(url)
                                .Build();

NOTE: I had to add nuget Microsoft.ApplicationInsights.AspNetCore

CAUTION: Disabling sampling might result throttling and too much network traffic in case of very high volume scenario. So you might want to look at Fixed rate sampling instead of adaptive. Refer Sampling in Application Insights.

Azure Application Insights - Not recording all requests on high traffic situations

1 Answers