.net httpclient on Azure w/ TLS/SNI: The request was aborted: Could not create SSL/TLS secure channel

Question

Similar questions have been asked multiple times before - but before flagging this one as duplicate please read on. Most of these questions are very old. I have worked through a lot of questions and answers and did not find a suitable solution.

We have an Azure Cloud Service project in .net 4.5. It connects to dozens of our customers' APIs (not necessarily cloud hosted) without any problems, but a single API fails with this error message:

The request was aborted: Could not create SSL/TLS secure channel

What am I missing here?

This is the code (slightly condensed) that I am using to connect to the API (this runs per API, so the base URL does not change):

ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol =
    SecurityProtocolType.Tls12 |
    SecurityProtocolType.Tls11 |
    SecurityProtocolType.Tls |
    SecurityProtocolType.Ssl3;

ApiClient = HttpClientFactory.Create();
ApiClient.DefaultRequestHeaders.Authorization = null;
ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", "Basic {passwordToken}");
ApiClient.DefaultRequestHeaders.Accept.Clear();
ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 50);

var sp = ServicePointManager.FindServicePoint(new Uri(baseUrl));
sp.SetTcpKeepAlive(true, 30000, 30000);

foreach (var request in urls)
{
    Result = new HttpResponseMessage();
    Result = await ApiClient.GetAsync(url);
    ...
}

This is what makes it hard to debug:

• This problem only occurs in production, i.e. when running as an Azure Cloud Service. Not when debugging locally.
• It only occurs with requests sent through HttpClient. Not with WebClient.
• Further research (comparing the APIs) revealed that this API is the only one that has enabled SNI and ONLY supports TLS1.2.

Suggestions considered from other questions/answers regarding SNI in .net Framework:

• To prevent misunderstandings: This is about the cloud service connecting to an API, not about a connection that is being made to the cloud service.
• The HttpClient instance is being reused for all requests to a single API. (This is important as this answer suggests that the SNI tag will be created with the domain HttpClient has been initialized with). I have also tried configuring TLS after the Factory instantiated the HttpClient. No change.
• The certificates are valid of course. No self-signed certificates but regular trusted ones off the shelf. Opening the API in any browser also works like a charm.
• TLS1.2 is not enabled by default in .net framework 4.5, but the line ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12should actually enable it. Is there anything wrong with the way I am doing it?
• Calling the API in curl (WSL and Azure remote bash) with curl --user 'user:pwd' https://myurl also works perfectly and returns the expected data.
• Testing tls1.2 with openssl like openssl s_client -connect hostname:443 -tls1_2 does not reveal any issues. The chain is displayed correctly, and a TLSv1.2 session is confirmed. Testing the server's SNI feature with openssl with openssl s_client -connect host:443 -tls1_2 -servername host -tlsextdebug -msg reveals SNI support by returning TLS server extension "server name" (id=0), len=0 I get the same certificate if I provide a completely different fantasy hostname though.
• I captured the TLS/SNI handshake when debugging locally (see screenshot below). No issues. My ability to debug ends with the cloud service. I would love to see the handshake between the cloud service and the API in WireShark, but I don't know of any option to analyze network traffic at that layer on an Azure cloud service. But if anyone knows how to capture the handshake process, I'd appreciate some hints.
• The server selects ECDHE-RSA-AES256-GCM-SHA384 as a cipher suite during the handshake with openssl which is pretty much default for TLS1.2. I don't have access to the cloud services list of cipher suites that it would provide at Client Hello - any idea how to find out?
• I don't have any proof that SNI is actually causing the problem, but this is the only difference between this API and dozens of others I can spot.

Stack trace:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at WCFServiceWebRole1.Controllers... in [the line calling GetAsync()]

Answer 1

I ended up re-creating a simple API on one of my servers and configured the software in such a way to send its requests there. That way I could capture the TLS handshake and analyze it in Wireshark. These are the supported cipher suites (client side, which is Azure cloud service):

And these are the cipher suites supported by the API which is not working:

I would assume that there should be a match so that server and client can agree on one. However, I cannot find a match... guess that is what is causing the problem. In fact, the list of supported cipher suites is much longer in a local debugging session - and there is at least one match which explains why it works locally.