write log to Application insights from local service fabric

Question

I am trying to integrate Azure App insights service into the service fabric app for logging and instrumentation. I am running fabric code on my local VM. I exactly followed the document here [scenario 2]. Other resources on docs.microsoft.com also seem to indicate the same steps. [ex: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-aggregation-eventflow For some reason, I don’t see any event entries in App insights. No errors in code when I do this:

ServiceEventSource.Current.ProcessedCountMetric("synced",sw.ElapsedMilliseconds, crc.DataTable.Rows.Count);

eventflowconfig.json contents


    {
      "inputs": [
        {
          "type": "EventSource",
          "sources": [
            { "providerName": "Microsoft-ServiceFabric-Services" },
            { "providerName": "Microsoft-ServiceFabric-Actors" },        
            { "providerName": "mystatefulservice" }
          ]
        }
      ],
      "filters": [
        {
          "type": "drop",
          "include": "Level == Verbose"
        }
      ],
      "outputs": [
        {
          "type": "ApplicationInsights",
          // (replace the following value with your AI resource's instrumentation key)
          "instrumentationKey": "XXXXXXXXXXXXXXXXXXXXXX",
          "filters": [
            {
              "type": "metadata",
              "metadata": "metric",
              "include": "ProviderName == mystatefulservice && EventName == ProcessedCountMetric",
              "operationProperty": "operation",
              "elapsedMilliSecondsProperty": "elapsedMilliSeconds",
              "recordCountProperty": "recordCount"
            }
          ]
        }
      ],
      "schemaVersion": "2016-08-11"
    }

In ServiceEventSource.cs

[Event(ProcessedCountMetricEventId, Level = EventLevel.Informational)]
    public void ProcessedCountMetric(string operation, long     elapsedMilliSeconds, int recordCount)
    {
        if (IsEnabled())
            WriteEvent(ProcessedCountMetricEventId, operation, elapsedMilliSeconds, recordCount);
    }

EDIT Adding diagnostics pipeline code from "Program.cs in fabric stateful service

using (var diagnosticsPipeline =
                ServiceFabricDiagnosticPipelineFactory.CreatePipeline($"{ServiceFabricGlobalConstants.AppName}-mystatefulservice-DiagnosticsPipeline")
            )
            {
                ServiceRuntime.RegisterServiceAsync("mystatefulserviceType",
                    context => new mystatefulservice(context)).GetAwaiter().GetResult();

                ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id,
                    typeof(mystatefulservice).Name);

                // Prevents this host process from terminating so services keep running.
                Thread.Sleep(Timeout.Infinite);
            }

In VS, Under View -> Other Windows select Diagnostic Events. This is a local ETW (EventSource) listener. You can check that events are getting emitted correctly. — Matt Trower - MSFT
Yes, Matt. I do see see the event in local ETW viewer. But, I want the events to show up in Azure app insights. Not local. Jeff, I don't see any logs on local that indicate errors in transmission to azure. — teeboy
That event source & event flow config all looks correct to me, however event sources can be finnicky. I've had good luck using this package in unit testing to help find issues with event sources. Once you've verified that the event source is properly configured and being called, I would suggest playing with that filter on the app insights output and see if that's potentially your issue. I've had experience where filters don't always work like one would expect. — pdylanross
@ pdylanross: The eventsourceconfig looks good. Do you think this is probably a firewall issue of some kind on Azure? Do I have to open ports or something for on-prem events to show up on Azure App insights? — teeboy

Diego Mendes Diego Mendes · Accepted Answer · 2018-03-08T10:41:57

Event Source is a tricky technology, I have been working with it for a while and always have problems. The configuration looks good, It is very hard to investigate without access to the environments, so I will make my suggestions.

There are a few catches you must be aware of:

If you are listening etw events from a different process, your process must be running with a user with permissions on 'Performance Log Users. Check which identity your service is running on and if it is part of performance log users, who has permissions to create event sessions to listen for these events.
Ensure the events are being emitted correctly and you can listen to them from Diagnostics Events Window, if it is not showing there, there is a problem in the provider.
For testing purposes, comment out the line if (IsEnabled()). it is an internal check to validate if your events should be emitted. I had situations where it is always false and skip the emit of events, probably it cache the results for a while, the docs are not clear how it should work.
Whenever possible, use the EventSource from the nuget package instead of the framework one, the framework version is full of bugs and lack fixes found in the nuget version.
Application Insights are not RealTime, sometimes it might take a few minutes to process your events, I would recommend to output the events to a console or file and check if it is listening correctly, afterwards, enable the AppInsights.

write log to Application insights from local service fabric

2 Answers