Service Fabric https endpoint with kestrel and reverse proxy

Question

I've been trying to setup Https on a stateless API endpoint following the instructions on the microsoft documentations and diverse post/blogs I could find. It works fine locally, but I'm struggling to make it work after deploying it on my dev server getting

Browser : HTTP ERROR 504
Vm event viewer : HandlerAsyncOperation EndProcessReverseProxyRequest failed with FABRIC_E_TIMEOUT
SF event table : Error while processing request: request url = https://mydomain:19081/appname/servicename/api/healthcheck/ping, verb = GET, remote (client) address = xxx, request processing start time = 2018-03-13T14:50:17.1396031Z, forward url = https://0.0.0.0:44338/api/healthcheck/ping, number of successful resolve attempts = 48, error = 2147949567, message = , phase = ResolveServicePartition

in code I have in the instancelistener

    .UseKestrel(options =>
    {
       options.Listen(IPAddress.Any, 44338, listenOptions =>
       {
           listenOptions.UseHttps(GetCertificate());
       });
    })

servicemanifest

<Endpoint Protocol="https" Name="SslServiceEndpoint" Type="Input" Port="44338" />

startup

        services.AddMvc(options =>
        {
            options.SslPort = 44338;
            options.Filters.Add(new RequireHttpsAttribute());
        });

+

            var options = new RewriteOptions().AddRedirectToHttps(StatusCodes.Status301MovedPermanently, 44338);
        app.UseRewriter(options);

here is what I got in azure (deployed through ARM template)

Health probes
NAME                    PROTOCOL    PORT    USED BY
AppPortProbe            TCP         44338   AppPortLBRule
FabricGatewayProbe      TCP         19000   LBRule
FabricHttpGatewayProbe  TCP         19080   LBHttpRule
SFReverseProxyProbe     TCP         19081   LBSFReverseProxyRule

Load balancing rules
NAME                    LOAD BALANCING RULE                 BACKEND POOL                    HEALTH PROBE
AppPortLBRule           AppPortLBRule (TCP/44338)           LoadBalancerBEAddressPool       AppPortProbe
LBHttpRule              LBHttpRule (TCP/19080)              LoadBalancerBEAddressPool       FabricHttpGatewayProbe
LBRule                  LBRule (TCP/19000)                  LoadBalancerBEAddressPool       FabricGatewayProbe
LBSFReverseProxyRule    LBSFReverseProxyRule (TCP/19081)    LoadBalancerBEAddressPool       SFReverseProxyProbe

I have a Cluster certificate, ReverseProxy Certificate, and auth to the api through azure ad and in ARM

                "fabricSettings": [
                {
                    "parameters": [
                        {
                            "name": "ClusterProtectionLevel",
                            "value": "[parameters('clusterProtectionLevel')]"
                        }
                    ],
                    "name": "Security"
                },
                {
                    "name": "ApplicationGateway/Http",
                    "parameters": [
                      {
                        "name": "ApplicationCertificateValidationPolicy",
                        "value": "None"
                      }
                    ]
                }
            ],

Not sure what else could be relevant, if you have any ideas/suggestions, those are really welcome

Edit : code for GetCertificate()

    private X509Certificate2 GetCertificate()
    {
        var certificateBundle = Task.Run(async () => await GetKeyVaultClient()
            .GetCertificateAsync(Environment.GetEnvironmentVariable("KeyVaultCertifIdentifier")));
        var certificate = new X509Certificate2();
        certificate.Import(certificateBundle.Result.Cer);
        return certificate;
    }

    private KeyVaultClient GetKeyVaultClient() => new KeyVaultClient(async (authority, resource, scope) =>
    {
        var context = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCred = new ClientCredential(Environment.GetEnvironmentVariable("KeyVaultClientId"),
            Environment.GetEnvironmentVariable("KeyVaultSecret"));
        var authResult = await context.AcquireTokenAsync(resource, clientCred);
        return authResult.AccessToken;
    });

Have you configured the endpoint binding in AplicationManifest? Other from that, if you login to your dev machine and examine the ports, do you see 44338 open? Another thing I'd recommend is to enable ConsoleRedirection logging in your ServiceManifest to see if anything meaningful will pop up there. And, of course, you could verify SF logs in Event Viewer. — Kiryl Z
Hello, thanks for reply, ports are open yes. according to microsoft doc (docs.microsoft.com/en-us/azure/service-fabric/…) that would be the EndpointCertificate and EndpointBindingPolicy? I'm not really sure to which certificate it refers, is it the clusters or the reverseproxy? One last thing I was concerned about is the ACLs on the server, I read it could be wrong, looking into how to check that and change it if necessary — Xav Sc
Refer to this example - matt.kotsenas.com/posts/https-in-service-fabric-web-api. You'll find there how you should tie your service in the AppManifest with the certificate. It's not about the reverseproxy though, it's about telling SF what cert it should bind the endpoint with. Also make sure you have your cert installed on the dev machine. — Kiryl Z
As the link doesn't speak at all about reverse proxy I suppose thats the clusters certificate. All the certs used are defined in the ARM template, so it "should" be at the right place already. I'll try all that, thanks for your help. — Xav Sc
still not working after adding to the appmanifest what was missing. I redeployed my cluster to set logs to verbose though I have this kind of messages "ERROR_WINHTTP_CANNOT_CONNECT" or "Re-resolved service url = 0.0.0.0:44338/api/healthcheck/ping". the 0.0.0.0:44338 looks kinda odd ... — Xav Sc

Kiryl Z Kiryl Z · Accepted Answer · 2018-03-15T08:43:56

Digging into your code I've realized that there is nothing wrong with it except one thing. I mean, as you use Kestrel, you don't need to set up anything extra in the AppManifest as those things are for Http.Sys implementation. You don't even need to have an endpoint in the ServiceManifest(although recommended) as all these things are about URL reservation for the service account and SSL binding configuration, neither of which is required with Kestrel.

What you do need to do is to use IPAddress.IPv6Any while you configure SSL. Aside the fact that it turns out to be the recommended way which allows you to accept both IPv4 and IPV6 connections, it also does a 'correct' endpoint registration in the SF. See, when you use IPAddress.Any, you'll get the SF setting up an endpoint like https://0.0.0.0:44338, and that's how the reverse proxy will try to reach the service which obviously wouldn't work. 0.0.0.0 doesn't correspond to any particular ip, it's just the way to say 'any IPv4 address at all'. While when you use IPAddress.IPv6Any, you'll get a correct endpoint mapped to the vm ip address that could be resolved from within the vnet. You could see that stuff by yourself in the SF Explorer if you go down to the endpoint section in the service instance blade.

Service Fabric https endpoint with kestrel and reverse proxy

1 Answers