HttpClient throwing "An error occurred while sending the request."

Question

I have three layer application architecture.

My Client --> My service A (REST hosted in IIS) --> Other Team's service X (REST).

Service A is ASP.Net 4.6.1 framework, not ASP.Net Core.

Client is communicating to A with HttpClient and A is communicating to X with HttpClient.

Client is firing almost 2500 calls to my service to A and to X.

Out of 2500 calls service A randomly (may be 10 calls) fails with below exception. Its not reproducible.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> 
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a 
receive. ---> System.IO.IOException: Unable to read data from the transport connection: An     
established connection was aborted by the software in your host machine. ---> 
System.Net.Sockets.SocketException: An established connection was aborted by the software in your 
host machine
at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags 
socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback 
callback, Object state)
 --- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace --

Here is my service A call. A in IIS calls below code block and its called by each request. X is taking user credentials and returning data based on user, so we are not sharing HttpClient between calls.

var user = (System.Security.Principal.WindowsIdentity)HttpContext.Current.User.Identity;
            System.Security.Principal.WindowsIdentity.RunImpersonated(user.AccessToken, () =>
      {
        static HttpClient Client = new HttpClient();
        static string CallX(string[] args)
        {
            HttpClientHandler handler = new HttpClientHandler
            {
                UseDefaultCredentials = true
            };    

            Client = new HttpClient(handler)
            {
                BaseAddress = new Uri("http://XserviceUrl/api/")
            };
            Client.Timeout = TimeSpan.FromSeconds(600);    
            var result = Client.PostAsync("Fake X controller"
                , new StringContent(JsonConvert.SerializeObject(args)
                , Encoding.UTF8, "application/json")).Result;    

            result.EnsureSuccessStatusCode();

            var json = result.Content.ReadAsStringAsync().Result;
            return DosomethingWithResult(json);    
        }
    });

Things I tried:

Some SO post suggested might be timeout issue. So I added 600 seconds in Client and in Service A. I also changed IIS request timeout from default 2 minutes to 10 (600 seconds).

That type of error is almost always a socket state error. If you run a Wireshark trace you will see the other side sending either a fin or rst first. It could be due to too much traffic, bad routing, or even a server side issue. To make sure it's the other side make sure all you outbounds are being done correctly, that you are not flooding the server and for any lingering sessions they've all been closed. Netstat can help you see stragglers. The most likely cause however is a bad network situation. The best place to look is on server side and finding out why the sessions are instantly rst — JWP
What about the other party "Team's service X", don't they seen any errors? — Frank Nielsen
X service is not seeing any errors. That's what makes it more difficult to reproduce or investigate. — user781700
Did you try making it full async instead of blocking async code? it could lead to deadlocks. And what was about the timeouts, did you check the timestamps of the failed requests? did they ran into timeouts or not? — Benjamin Schäublin
Don’t create a new HttpClient upon each call, this is the very likely cause of this issue. HttpClient is supposed to be created once and reused for life of your app. At most, create just a few of them. aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong — Roman Polunin

user781700 user781700 · Accepted Answer · 2020-11-11T21:46:35

After some digging we resolved the issue. When A was sending the request to X, A was setting the Connection: keep-alive and the X was responding with Connection: Close property in header.

So after some calls A was exhaust with opened tcp connection and randomly it was throwing the error.

(Fiddler helped me to figure this out)

So all we had to do was setting ConnectionClose property of HttpClient

_client.DefaultRequestHeaders.ConnectionClose = true;

HttpClient throwing "An error occurred while sending the request."

2 Answers