HTTPS GRPC connections from C core clients to .NET core server: TLS handshake failure

Question

This question is closely related to one I asked yesterday, but my diagnostic information is different enough that I thought I'd update and resubmit: let me know if I should delete one of these.

I have a toy GRPC server written in .NET core that I need to connect to using a client using the grpc C core.

You can find the Startup.cs and Program.cs for the .net core server here. Nothing too interesting, except a call to UseHttps. I've verified the server works by connecting to it (over https) from a .net core client.

However I've tried to connect to this server from clients written in both C++ and python now, and the result is a GRPC error 14 and the following message on the client side

E0304 08:36:58.510065400 4905 ssl_transport_security.cc:1455] Handshake failed with fatal error SSL_ERROR_SSL: error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE.

and the following message on the server side.

dbug: Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer[2] 
  Connection id "0HM6UG4PBICBP" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel[1]
      Connection id "0HM6UG4PBICBP" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090367): No common application protocol exists between the client and the server. Application protocol negotiation failed.

The C++ client loads the server's public key into an SslOpts.pem_root_certs and uses that to create SslCredentials and then Channel objects. The Python client just doesn't read the cert file at all, and creates its channel as grpc.secure_channel("localhost:50052", grpc.ssl_channel_credentials()). I don't think it matters though: the TLS handshake fails before any cert information is exchanged.

I used wireshark to see if there was anything informative in the packets. From there I can see that the .NET core client sends this TLS client hello, which elicits a corresponding server hello. The client hello sent by the Python client looks like this (C++ packet is similar), and the server immediately responds with a handshake failure message.

My knowledge of TLS is rudimentary, but a couple of differences between the .NET and Python hellos stood out to me.

The working TLS hello packet seems to be TLS 1.2 "all the way through." The start of the TLS sequence looks like this

 Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 160
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 156
            Version: TLS 1.2 (0x0303)

Whereas the corresponding sequence in the failed TLS packet is

Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 512
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 508
            Version: TLS 1.2 (0x0303)

For some reason, there's a mention of TLS 1.0 in the middle there

In the server hello the server chooses the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) cipher suite, which isn't mentioned in the failed TLS hello packet.

There are other differences of course, but to my untrained eye they seem less relevant.

I have a couple of specific questions beyond "how do I get this to work?"

Why is the grpc client complaining about something related to ssl 3 when I can see that it's sending its hello over tls 1.2 (with that reference to tls 1.0)?
What are the real differences between the two client hello packets? I can see the differences in cipher suite, the TLS version thing I mentioned earlier, that each has some extensions that the other doesn't: but I don't know what's relevant here, especially as it pertains to why the asp.net core server would reject the client hello
Is there some grpc C-core environment variable I should be setting to solve this, and if so how? I've tried launching the client with GRPC_SSL_CIPHER_SUITES=ECDHE-RSA-AES256-SHA384 python client.py (having looked up the conversion here), but I just get yelled at with the following error

E0304 08:13:28.257362600 4893 ssl_transport_security.cc:815] Invalid cipher list: ECDHE-RSA-AES256-SHA384.

The python client sending that failing packet is run via Python 3, Ubuntu 18.04 (on WSL)

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017 (Library: OpenSSL 1.1.1  11 Sep 2018)

bumbling-tadpole bumbling-tadpole · Accepted Answer · 2021-03-12T00:21:28

My connection troubles seem to have been caused by some misconfiguration on my machine, as I was unable to reproduce the errors on other setups. So that settles the "how do I get this to work?"

In response to those three specific questions I answered.

The helpful folks on my grpc issue came to the conclusion that no, the mixing of TLS versions I mentioned is not a problem
I'm still not sure what is causing one client hello packet to be rejected and the other accepted, but again it seems to be something specific to my machine which is not a problem on fresh Windows server 2019 or windows 10 VMs I've created
Not sure about this one: I'm pretty sure that I'm setting that environment variable correctly but I'm not sure why openssl doesn't recognize that value. Also, I don't think GRPC_SSL_CIPHER_SUITES has any effect on windows

HTTPS GRPC connections from C core clients to .NET core server: TLS handshake failure

1 Answers