ASP.NET HttpWebRequest stop sending requests, high CPU usage

Question

I have an old ASP.NET application. We send out httpWebRequest to a remote REST server and retrieve XML back, most of the time the application works fine. Recently, we got some high CPU usage issue several times a day.

During the high CPU usage time, we monitored those httpWebRequest connections (by checking netstat for the w3wp process). At the very beginning, the connections change to "CLOSE_WAIT" status from "ESTABLISHED", then after those connections timeout, those connections disappeared one by one, and then there is no connection any more.

After resetting the IIS, when the w3wp.exe process start again, we still could not find any connections to httpWebRequest target server. So the CPU usage keep staying at high level. Even after several round of reset, it won't solve the issue, until we saw some connections start to connect to httpWebRequest target server, the CPU usage went down.

I actually thought it could be the issue of my code not handling the httpWebRequest properly, I posted another question here: How to close underlying connections after catch httpwebrequest timeout?.

As mentioned in that question, I also found lots of timeout exceptions for System.Net.HttpWebRequest.GetResponse(). We found 3500 of the same exception within 5 minutes when CPU usage is really high.

What could cuase this type of issue and what could be the medicine? Why won't the application send out request any more (since there is no connection in netstat)?

Here is the source code just in case:

System.Net.HttpWebResponse httpWebResponse = null;
System.IO.Stream stream  = null;
XmlTextReader xmlTextReader  = null;
try
{
    System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(request);
    httpWebRequest.ReadWriteTimeout = 10000;
    httpWebRequest.Timeout = 10000;
    httpWebRequest.KeepAlive = false;
    httpWebRequest.Method = "GET";
    httpWebResponse = (System.Net.HttpWebResponse)httpWebRequest.GetResponse();
    stream = httpWebResponse.GetResponseStream();
    xmlTextReader = new  XmlTextReader(stream);
    xmlTextReader.Read();
    xmlDocument.Load(xmlTextReader);
    //Document processing code.
    //...
}
catch
{
    //Catch blcok with error handle
}
finally
{
    if (xmlTextReader != null)
        xmlTextReader.Close();
    if (httpWebResponse != null)
        httpWebResponse.Close();
    if (stream != null)
        stream.Close();
}

Answer 1

I would suggest you to try sending requests asynchronously to avoid blocking the main thread:

using (var client = new WebClient())
{
    client.OpenReadCompleted += (sender, e) =>
    {
        using (var reader = XmlReader.Create(e.Result))
        {
            // Process the XML document here
        }
    };
    client.OpenReadAsync(new Uri("http://www.example.com"));
}

Answer 2

From your description, it is not clear to me that your high CPU utilization is related to your outgoing HTTP requests. High CPU utilization could be due to a bug in your code, a bug in CLR, IIS, or something else. Without knowing which component is consuming the CPU, you wont be able to do anything further.

If I were you, I would first try to attach a sampling profiler to the W3WP process, and see which component is consuming the CPU. That should point you to the next steps in resolving this issue.

Answer 3

Finding the reason for the high CPU utilization can take some time since you will have to locate the code that is causing the problem. I am working through this right now on a vb.net app that I recently developed. In the meantime, I have developed a page that has a button which an Administrative user can click to stop the W3WP.exe process. It is a great stop gap measure until the problem code can be identified and updated. Here is the code I used. Just create a .aspx page with a button that call the following code on the corresponding .aspx.vb page. The code uses the command prompt to get the Tasklist and writes this to a file. I then parse the text file for the PID of the W3WP.exe worker process. Then, I access the command prompt programmatically again to terminate the W3WP.exe process using the PID.

Imports System.Web

Partial Class TAP
Inherits System.Web.UI.Page

Protected Sub btnStop_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnStop.Click
    Call thStop_IIS()
End Sub

Function thStop_IIS()

    Dim varRetval As String
    Dim varPID As String
dim x as long
    Dim savePath As String = Request.PhysicalApplicationPath + "exports\"
    If Dir(savePath + "filename.txt") = "filename.txt" Then
        Kill(savePath + "filename.txt")
    End If

    varRetval = Shell("cmd /c tasklist > " + savePath + "filename.txt")

    For x = 1 To 90000000

    Next x

    varPID = thParse_File_Return_PID(savePath + "filename.txt")
    varRetval = Shell("cmd /c taskkill /pid " & varPID & " /F")
    Return True
End Function

Function thParse_File_Return_PID(ByVal varFileToParse As String) As Integer

    On Error GoTo handler_err

    'Dim FILE_NAME As String = "C:\Users\Owner\Documents\test.txt"
    Dim FILE_NAME As String = varFileToParse
    Dim TextLine As String
    Dim varPID As Integer
    Dim x As Long

    If System.IO.File.Exists(FILE_NAME) = True Then

        Dim objReader As New System.IO.StreamReader(FILE_NAME)

        Do While objReader.Peek() <> -1
            'TextLine = TextLine & objReader.ReadLine() & vbNewLine
            TextLine = objReader.ReadLine() & vbNewLine
            If InStr(TextLine, "w3wp.exe") Then
                varPID = Mid(TextLine, 31, 4)
            End If
        Loop

        thParse_File_Return_PID = varPID

    Else
        thParse_File_Return_PID = 0
    End If



handler_exit:
    Exit Function

handler_err:
    'MsgBox(Err.Number & " " & Err.Description & ":" & "thParse_File_Return_Pages")
    Resume

 End Function

End Class