Missing endpoints in endpoint based routing .net core 3.1

Question

I'm working with multitenant .net core web application which consists of WebApi and MVC routing. With normal behavior application should go to fallback controller, where using external help will be decided on which of master/tenant controller will be performed redirect.But sometimes application refuses to find proper fallback to action. After some investigation, I've found that at these moments, not all endpoints are being registered. After two days of search, I figured out that most of these problems affect Razor, but none of them are applicable to MVC/WebApi. I start to think that this is caused by having second branch with .MapWhen() but its not confirmed.

So I'd appreciate any help on solving this problem. Also attaching my current configuration of routing down below:

ConfigureServices method:

services.AddControllersWithViews(options =>
    {
        options.Filters.Add(typeof(ReverseProxyFilter));
        options.Conventions.Add(new ApiExplorerGroupPerVersionConvention());
    })
    .AddRazorRuntimeCompilation()
    .AddApplicationPart(typeof(EmailNamespace).Assembly)
    .AddViewLocalization()
    .SetCompatibilityVersion(CompatibilityVersion.Latest)
    .AddNewtonsoftJson(options =>
        {
        options.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
        options.SerializerSettings.DateParseHandling = DateParseHandling.DateTimeOffset;
        options.SerializerSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
        })
    .AddControllersAsServices();

services.AddRazorPages();

services.Configure<ForwardedHeadersOptions>(options =>
    {
        options.ForwardedHeaders = ForwardedHeaders.XForwardedProto;
    });

services.Configure<RouteOptions>(routeOptions =>
    {
        routeOptions.ConstraintMap.Add("master", typeof(MasterRouteConstraint));
        routeOptions.ConstraintMap.Add("tenant", typeof(TenantRouteConstraint));
    });

Configure method:

app.UseMiddleware<TenantFilterMiddleware>();

app.UseHttpStatusCodeExceptionMiddleware();

app.UseResponseCaching();

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();

app.UseEndpoints(endpoints =>
{
    endpoints.MapHub<WebsocketsHub>("/hubs");
    endpoints.MapControllers();
    endpoints.MapRazorPages();
});

app.MapWhen(
    context => !context.Request.Path.Value.StartsWith("/api"),
    builder =>
        {
            builder.UseRouting();
            builder.UseEndpoints(endpoints =>
            {
                endpoints.MapFallbackToController("Index", "Fallback");
            });
    });

Example of exception:

An unhandled exception occurred while processing the request.
InvalidOperationException: Cannot find the fallback endpoint specified by route values: 
{ 
    action: Index,
    controller: Fallback,
    area: 
}.
Microsoft.AspNetCore.Mvc.Routing.DynamicControllerEndpointMatcherPolicy.ApplyAsync(HttpContext HttpContext, CandidateSet candidates)

What do you mean by registering? What protocol is being used in Internet? Is it TCP? I would start by using from cmd.exe >Netstat -a on controller and check the listeners and connections by port number. You will only send data on clients that are connected. So first thing is to find out which clients are connected and compare to the ones that are registered. I not that familiar with MVC but do know internet routing very well. We need to find out if the URL are correct. — jdweng
@jdweng, by registering I mean: Application will have dictionary for endpoints after application start. When all started correctly this dictionary has about ~480 endpoints of different Types, but sometimes, it only has ~240 endpoints all of them of same Type. I believe that URL are correct since request reach server 100% of the time, so my assumption this is more configuration related issue then basic internet routing problem. — Hephaestus901
How does endpoints get removed when server goes off line? How does code determine when a fallback endpoint should get used? Have you looks at sniffer results like wireshark or fiddler? I would assume an attempt to connect to a controller (maybe an ARP) can be seen and when no response then another attempt is made to fallback. A machine has an ARP table which you can see from cmd.exe >ARP -A which will give IP address of machines seen recently. Usually each machine ARP once every 15 minutes and then if not seen after 1/2 hour get removed from ARP table. — jdweng
One theory could be that Controller went offline recently but still in ARP table. — jdweng
@jdweng 1) As far as I understand .net routing, endpoints cannot be removed at all once the endpoints table was built. 2) Fallback endpoint work like last resort endpoint if non of existing endpoints are not valid for given route values, in my situation fallback configured at the end of second code block with following line: endpoints.MapFallbackToController("Index", "Fallback"); so it will scan all existing endpoints, and if none fits, it should go to Fallback controller's Index action. but this actions is not being registered as endpoint somehow — Hephaestus901

Hephaestus901 Hephaestus901 · Accepted Answer · 2020-07-09T19:32:12

Well after two days of trying and researching, the solution was dead simple as usual. The cure for this problem was slightly rewriting routing pipeline. So as far as I found out, microsoft routing system highly prefer branches to be placed before default branch and not after, in this case, all system behavior starts working as expected in 100% of cases.

So instead of having this:

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();

app.UseEndpoints(endpoints =>
{
    endpoints.MapHub<WebsocketsHub>("/hubs");
    endpoints.MapControllers();
    endpoints.MapRazorPages();
});

app.MapWhen(
    context => !context.Request.Path.Value.StartsWith("/api"),
    builder =>
        {
            builder.UseRouting();
            builder.UseEndpoints(endpoints =>
            {
                endpoints.MapFallbackToController("Index", "Fallback");
            });
    });

Now I have this:

app.MapWhen(context => !context.Request.Path.Value.StartsWith("/api"), ConfigureApiPipeline);

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.UseSession();

app.UseEndpoints(MapBasicEndpoints);

Where ConfigureApiPipeline() function looks like this:

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.UseSession();
app.UseEndpoints(endpoints =>
    {
        MapBasicEndpoints(endpoints);
        endpoints.MapHealthChecks(SystemLivenessCheck.Path, new HealthCheckOptions()
                {
                    AllowCachingResponses = false,
                    ResultStatusCodes = new Dictionary<HealthStatus, int>
                    {
                        [HealthStatus.Healthy] = StatusCodes.Status200OK,
                        [HealthStatus.Degraded] = StatusCodes.Status503ServiceUnavailable,
                        [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
                    },
                });
        endpoints.MapFallbackToController("Index", "Fallback");
    });

And MapBasicEndpoints is:

private static void MapBasicEndpoints(IEndpointRouteBuilder endpoints)
{
    endpoints.MapHub<WebsocketsHub>("/hubs");
    endpoints.MapControllers();
}

Missing endpoints in endpoint based routing .net core 3.1

1 Answers