Organizing Application Insights / Health checking for single-tenant services

Question

I'm managing a product where we setup a complete environment for each customer. The product consists of an Angular frontend, with an ASP.Net.Core API backend and a SQL server database.

Resulting architecture is that we have e.g.

https://customer1.product.com and https://customer1.product.com/api

https://customer2.product.com and https://customer1.product.com/api

etc... A site for each customer, but multiple customer sites share same IIS server.

I've been looking into enabling Application Insight at least for the API, and enable Health Checking.

I got both of these working in proof-of-concept - the question here is how I best could organize a long list of sites in Application Insights.

I would like to get total Insight into the server, with summed up total number of request across API's, etc.

This leads me to one shared Insight Resource, and all the API's share the same instrumentation key.

For health checking though, I can see I can only add 100 ping tests to one resource. I was considering establishing an independent service, that, using a list of sites, could use the new .Net core 2.2. Health Checking and ping each site from an external server - and then setup this on .Net Core Health Check as ping test on the Insight Resource. (Used this as inspiration: https://www.hanselman.com/blog/HowToSetUpASPNETCore22HealthChecksWithBeatPulsesAspNetCoreDiagnosticsHealthChecks.aspx). But then I would not benefit from the Insights regional tests - as all request would come from my Health Check API.

But I am unsure if and how I could set this up, so that once a single site goes down, e.g. by misconfiguration, I would be notified that only one site is down.

So I would like some input on how others, if any, has implemented scenarios like this.

I would like

• Insight on a server level, e.g. requests, load, etc combined for sites.
  • But drill down to a single site would also be nice to split load for customers
• Two ping tests on each site, Angular index.html and API call that connects to database.
• Easy setup, so when we get new customers, the monitoring part would be either self-configuring or scriptable. At least so we shouldn't log into Azure Insights portal.

I am not necessarily looking for complete implementation details, more architectural guidelines on how I could structure a setup like this.

Suggestions for other products than Application Insights are welcome as well. Just saw Insights as a good fit for Asp.Net.Core API. Already using Sentry to gather error reports.

Best regards /Anders

Answer 1

Let me share what's possible, hope it will help with a solution.

Quick note - it is possible to script everything through ARM if needed. Application Insights resources included. Should not be a concern whatever solution you pick at the end (different solutions might require more scripting though).

Option 1. Every one gets its own AI resource. All telemetry (server, client, availability) gets reported to it.

This will get you nice alerting capabilities, drill into experiences. But it will be hard to use curated experiences for across customer queries.

Having said it - there is actually a way to do across Application Insights resource queries in Application Insights Analytics. It is hard to make such queries efficient (since they potentially need to pass data across clusters), so you need to play to understand whether you can get what you need.

Option 2. Have one AI resource for all telemetry for all customers. Customer is differentiated by Operation Name only.

Easy to do across customer queries. But harder to do investigation for a particular customer. Also there is a limitation of 100 web tests per resource

Option 3. Have one AI resource for server/client telemetry. Use RoleName to differentiate customers. Have AI resource / customer for Availability.

This allows to get both summary data across customers and data scoped to a single RoleName (customer).

Availability - allows to configure needed number of tests per customer. Have individual report per customer. Distributed tracing should continue to work (Application Insights can work across resources).

Note - this works only if your topology is simple and you don't need to use RoleName for something else.

Hope it helps!