Pipe All AKS microservices into a single Application Insights and be able to filter by microservice name

2
votes

I would like to know if it is possible to have multiple Azure Kubernetes micro services pointing to the same Application Insights Resources and be able to use Log Queries to split the logs per micro services name.

At the moment, I have been able to create an Application Insights resource. I also configured the micro services to point to that Application Insights resource using Visual Studio 2017 IDE (all done via a GUI).

That being said, I now have all my micro services linked to the same Application Insights. I would like to be able to filter my log queries by micro service.

The reason behind this is that the company I work for might soon have a lot of micro services and do not want to manage N application insights resources. Instead, we want everything centralized in the same Application Insights (so I cannot use the union operator to join multiple application insights since we only want to have one).

I thought about adding custom fields to Application Insights, but it does not seem like it can help in my situation. https://docs.microsoft.com/en-us/azure/azure-monitor/platform/custom-fields

Thanks!

UPDATE: Sample code: Telemetry Clas Program.cs: enter image description here Custom dimensions: enter image description here

After reading this post: Adding Custom Dimension to Request Telemetry - Azure functions I realized that I need to call TrackEvent(), TrackTrace(), etc. in order to see the new custom dimensions entry. However, I needed to add a Microservice column to any request/event/trace produced by a given microservice.

2
azurekubernetesmicroservicesazure-application-insights

2 Answers

3
votes

The reason behind this is that the company I work fore might soon have a lot of micro services and do not want to manage N application insights resources. Instead, we want everything centralized in the same Application Insights (so I cannot use the union operator to join multiple application insights since we only want to have one)

Do mind that App Insights price model is mainly based on cost per GB of data ingested. The first couple of GB per month is free. So your solution might not be the most cost efficient.

I thought about adding custom fields to Application Insights, but it does not seem like it can help in my situation. https://docs.microsoft.com/en-us/azure/azure-monitor/platform/custom-fields

For App Insights you could use custom properties. There are several ways to set them (using code!). One is to set them directly as shown here:

using Microsoft.ApplicationInsights.DataContracts;

var gameTelemetry = new TelemetryClient();
gameTelemetry.Context.GlobalProperties["Game"] = currentGame.Name;
// Now all telemetry will automatically be sent with the context property:
gameTelemetry.TrackEvent("WinGame");

Do mind that this method will only add properties to the telemetry created using the TelemetryClient instance!

the other one is using a telemetry initializer:

using System;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;

namespace MvcWebRole.Telemetry
{
  /*
   * Custom TelemetryInitializer that overrides the default SDK
   * behavior of treating response codes >= 400 as failed requests
   *
   */
  public class MyTelemetryInitializer : ITelemetryInitializer
  {
    public void Initialize(ITelemetry telemetry)
    {
        var requestTelemetry = telemetry as RequestTelemetry;
        // Is this a TrackRequest() ?
        if (requestTelemetry == null) return;
        int code;
        bool parsed = Int32.TryParse(requestTelemetry.ResponseCode, out code);
        if (!parsed) return;
        if (code >= 400 && code < 500)
        {
            // If we set the Success property, the SDK won't change it:
            requestTelemetry.Success = true;

            // Allow us to filter these requests in the portal:
            requestTelemetry.Properties["Overridden400s"] = "true";
        }
        // else leave the SDK to set the Success property
    }
  }
}

I prefer this one as it will also apply to telemetry created by the framework, not only to custom telemetry created using your own code.

I would like to know if it is possible to have multiple Azure Kubernetes micro services pointing to the same Application Insights Resources and be able to use Log Queries to split the logs per micro services name.

Now, there might already be something in place. All telemetry have properties like cloud_Rolename & cloud_RoleInstance:

requests
| project cloud_RoleInstance , cloud_RoleName

it will show the container name and pod name so that might already be good enough for you.

So, for example, this will show requests split by pod:

requests
| summarize count() by cloud_RoleName

If you use C# you could also add this package created by Microsoft that will attach a lot of AKS details to the telemetry like pod id, pod labels, replica set name etc.

1
votes

I have been able to fix the issue by adding some steps to the solution Peter gave me.

  1. I created a custom telemetry initializer that looks like the following:
 public class MicroserviceNameTelemetry : ITelemetryInitializer
    {
        private string microserviceName;
        public MicroserviceNameTelemetry(string microserviceName)
        {
            this.microserviceName = microserviceName;
        }
        public void Initialize(ITelemetry telemetry)
        {
            telemetry.Context.GlobalProperties["Microservice"] = microserviceName;
        }

    }

I then had to overwrite the default telemetry initializer in the Startup.cs (the step I added):

public static IServiceCollection AddApplicationInsights(this IServiceCollection services, IConfiguration configuration)
{
            services.AddApplicationInsightsTelemetry(configuration);
            services.AddSingleton<ITelemetryInitializer, MicroserviceNameTelemetry>((serviceProvider) => {
                return new MicroserviceNameTelemetry(configuration.GetSection("Microservice").GetValue<string>("name"));
            });
            // Enable K8s telemetry initializer            
           services.AddApplicationInsightsKubernetesEnricher();

            return services;
}