Why is Application Insights Preventing my Worker Role from Starting?

Question

I have an Azure Cloud Service with a single worker role that will deploy, start, and run without fail every time, but I ran into some intermittent problems I wanted to diagnose with Application Insights. I followed directions from this article as well as here.

After installing the nugget packages and adding the one line of code, I ran the cloud service locally in both debug and release mode, and was able to see the AI info being reported to the Application Insights resource. So I packaged it up and uploaded the new configuration to my cloud service.

But then from the cloud the worker role will not actually start--it goes into a recycle death spiral, and in the "Diagnose and Solve Problems" blade I get a variation of the error below, always about "System.Threading.AsyncLocal'1'" and not being able to load the RoleEntryPoint. Following this article did not shed much light onto things, as the error pretty much tells me why it keeps recycling but not any clues on what to do about it.

Production - WebReportDownloader_IN_0: BusyRole Waiting for role to start... Failed to load role entrypoint. System.TypeLoadException: Could not load type 'System.Threading.AsyncLocal`1' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. at Microsoft.ApplicationInsights.Extensibility.SdkInternalOperationsMonitor.Exit() at Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(TelemetryConfiguration configuration, TelemetryModules modules, String serializedConfiguration) at Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active() at WebReportDownloader..ctor() in C:\Users\User\Source\Repos\Program\WebReportDownloader\WebReportDownloader.cs:line 21 --- End of inner exception stack trace --- at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck) at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.Activator.CreateInstance(Type type, Boolean nonPublic) at System.Activator.CreateInstance(Type type) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) [2017-09-14T15:28:21Z] Last exit time: [2017/09/14, 15:28:21.944]. Last exit code: 0.

Normally I can get enough of a hint from searching the exception and/or details, but this time I can't find anything that resembles my problem. All I can guess is that something small but critical is awry with my configuration, but I haven't been able to determine where I've gone off the rails based on the instructions I linked above.

My ServiceConfiguration.Cloud file...

    <?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="WebReportDownloader" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="4" osVersion="*" schemaVersion="2015-04.2.6">
  <Role name="WebReportDownloader">
    <Instances count="2" />
    <ConfigurationSettings>
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="admin" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="ENCRYPTEDVALUE" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="2018-09-01T23:59:59.0000000-04:00" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="true" />
      <Setting name="APPINSIGHTS_INSTRUMENTATIONKEY" value="KEYVALUE" />
    </ConfigurationSettings>
    <Certificates>
      <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption" thumbprint="847EEDE0084CF57A5A774CAE9E700713726CC856" thumbprintAlgorithm="sha1" />
    </Certificates>
  </Role>
</ServiceConfiguration>

And my ApplicationInsights.config...

<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
    <TelemetryInitializers>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.HttpDependenciesParsingTelemetryInitializer, Microsoft.AI.DependencyCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureWebAppRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.BuildInfoConfigComponentVersionTelemetryInitializer, Microsoft.AI.WindowsServer"/>
    </TelemetryInitializers>
    <TelemetryModules>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector">
            <ExcludeComponentCorrelationHttpHeadersOnDomains>
                <!-- 
        Requests to the following hostnames will not be modified by adding correlation headers. 
        This is only applicable if Profiler is installed via either StatusMonitor or Azure Extension.
        Add entries here to exclude additional hostnames.
        NOTE: this configuration will be lost upon NuGet upgrade.
        -->
                <Add>core.windows.net</Add>
                <Add>core.chinacloudapi.cn</Add>
                <Add>core.cloudapi.de</Add>
                <Add>core.usgovcloudapi.net</Add>
                <Add>localhost</Add>
                <Add>127.0.0.1</Add>
            </ExcludeComponentCorrelationHttpHeadersOnDomains>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
            <!--
      Use the following syntax here to collect additional performance counters:

      <Counters>
        <Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Handle Count" ReportAs="Process handle count" />
        ...
      </Counters>

      PerformanceCounter must be either \CategoryName(InstanceName)\CounterName or \CategoryName\CounterName

      NOTE: performance counters configuration will be lost upon NuGet upgrade.

      The following placeholders are supported as InstanceName:
        ??APP_WIN32_PROC?? - instance name of the application process  for Win32 counters.
        ??APP_W3SVC_PROC?? - instance name of the application IIS worker process for IIS/ASP.NET counters.
        ??APP_CLR_PROC?? - instance name of the application CLR process for .NET counters.
      -->
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule, Microsoft.AI.PerfCounterCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.DeveloperModeWithDebuggerAttachedTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnhandledExceptionTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnobservedExceptionTelemetryModule, Microsoft.AI.WindowsServer">
            <!--</Add>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.FirstChanceExceptionStatisticsTelemetryModule, Microsoft.AI.WindowsServer">-->
        </Add>
    </TelemetryModules>
    <TelemetryProcessors>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryProcessor, Microsoft.AI.PerfCounterCollector"/>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.AutocollectedMetricsExtractor, Microsoft.ApplicationInsights"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
            <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
            <ExcludedTypes>Event</ExcludedTypes>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
            <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
            <IncludedTypes>Event</IncludedTypes>
        </Add>
    </TelemetryProcessors>
    <TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>
<!-- 
    Learn more about Application Insights configuration with ApplicationInsights.config here: 
    http://go.microsoft.com/fwlink/?LinkID=513840

    Note: If not present, please add <InstrumentationKey>Your Key</InstrumentationKey> to the top of this file.
  --></ApplicationInsights>

And my "OnStart()" method where I actually set the InstrumentationKey...

public override bool OnStart()
        {
            // Set the maximum number of concurrent connections
            ServicePointManager.DefaultConnectionLimit = 12;
            TelemetryConfiguration.Active.InstrumentationKey = RoleEnvironment.GetConfigurationSettingValue("APPINSIGHTS_INSTRUMENTATIONKEY");

            // For information on handling configuration changes
            // see the MSDN topic at https://go.microsoft.com/fwlink/?LinkId=166357.

            bool result = base.OnStart();

            Trace.TraceInformation("WebReportDownloader has been started");

            return result;
        }

I would appreciate any insight (ha!) or guidance anyone can provide...

Update:

Through further experimentation, I've determined that whenever I'm trying to call the Application Insights assembly directly (such as when creating a Telemetry Client or setting the ApplicationInsights Resource Key directly) or when the ApplicationInsight TraceListener is called because of a trace the exception below is called

Could not load type 'System.Threading.AsyncLocal`1' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.

If I do the run-time instrumentation, I'm able to get some basic traces and logs to my Insight resources, but attempting to get anything more still escapes me.

Additionally, I'm currently targeting .NET 4.6.1 (information I neglected to mention previously).

Answer 1

After much back and forth with Azure support, I was finally able to get a screen sharing session set up. The helpful gentleman zeroed in on the problem almost immediately: Because my worker role project targets .NET 4.6.1 the ServiceConfiguration files (both .Cloud.cscfg and .Local.cscfg) need to be targeting osFamily 5.

The defaults generated by the VS17 wizard set it at osFamily=4 as seen below...

However, what I needed to make everything run properly was to be targeting osFamily=5 as seen below...

That one small change allowed the web role to deploy properly while instantiating a TelemetryClient.

Answer 2

According to your description, I created my Azure Cloud Service project and added the single Worker Role targets on .NET Framework 4.5, then installed Microsoft.ApplicationInsights.WindowsServer 2.4.1 and just added the code as you mentioned for setting the instrumentation key as follows:

WorkerRole.cs

public class WorkerRole : RoleEntryPoint
{
    private readonly CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
    private readonly ManualResetEvent runCompleteEvent = new ManualResetEvent(false);

    public override void Run()
    {
        Trace.TraceInformation("WorkerRoleB is running");

        try
        {
            this.RunAsync(this.cancellationTokenSource.Token).Wait();
        }
        finally
        {
            this.runCompleteEvent.Set();
        }
    }

    public override bool OnStart()
    {
        // Set the maximum number of concurrent connections
        ServicePointManager.DefaultConnectionLimit = 12;

        TelemetryConfiguration.Active.InstrumentationKey = RoleEnvironment.GetConfigurationSettingValue("APPINSIGHTS_INSTRUMENTATIONKEY");

        // For information on handling configuration changes
        // see the MSDN topic at https://go.microsoft.com/fwlink/?LinkId=166357.

        bool result = base.OnStart();

        Trace.TraceInformation("WorkerRoleB has been started");

        return result;
    }

    public override void OnStop()
    {
        Trace.TraceInformation("WorkerRoleB is stopping");

        this.cancellationTokenSource.Cancel();
        this.runCompleteEvent.WaitOne();

        base.OnStop();

        Trace.TraceInformation("WorkerRoleB has stopped");
    }

    private async Task RunAsync(CancellationToken cancellationToken)
    {
        // TODO: Replace the following with your own logic.
        while (!cancellationToken.IsCancellationRequested)
        {
            Trace.TraceInformation("Working");
            await Task.Delay(1000);
        }
    }
}

After deployed to Azure Cloud Service, it could work as expected and I could retrieve the diagnostic data from AI as follows:

Note: As Install the SDK in each project states that you need to set the ApplicationInsights.config file to be copied always to the output directory.

Additionally, you could refer to the quick start and instruments your app at run time to narrow this issue. Moreover, if I missed something or you did some other special code, you could update your question for us to troubleshoot this issue.