What is limiting the # of simultaneous connections my ASP.NET application can make to a web service?

Question

I have an ASP.NET 4.0 application running atop IIS 7.5 on a 64-bit Windows Server 2008 R2 Enterprise machine with gobs of RAM, CPU, disk, etc.

With every web request, the ASP.NET application makes a connection to a backend web service (via raw sockets), which is running on the same machine.

Problem: There appears to be something limiting the # of simultaneous connections to the backend web service. Suspiciously, the number of concurrent connections is topping out at 16.

I found this key article from Microsoft explaining how to tweak IIS' settings to accomodate ASP.NET apps that make lots of web service requests: http://support.microsoft.com/?id=821268#tocHeadRef

I followed the article's recommendatinos, but still no luck. The setting that is particularly interesting is the maxconnection setting, which I even bumped to 999.

Any ideas what else could be throttling connections?

Note: When I cut IIS out of the mix and have the clients connect directly to the backend web service, it will happily open as many connections as I need, so I'm positive the backend is not the bottleneck. It must be something in IIS/ASP.NET-land.

Here's the relevant section of the machine.config which I'm sure is being read by the application (verified with appcmd.exe):

<system.web>
    <processModel autoConfig="false" maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />
    <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152"/>

    <httpHandlers />

    <membership>
        <providers>
            <add name="AspNetSqlMembershipProvider"
                type="System.Web.Security.SqlMembershipProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="LocalSqlServer"
                enablePasswordRetrieval="false"
                enablePasswordReset="true"
                requiresQuestionAndAnswer="true"
                applicationName="/"
                requiresUniqueEmail="false"
                passwordFormat="Hashed"
                maxInvalidPasswordAttempts="5"
                minRequiredPasswordLength="7"
                minRequiredNonalphanumericCharacters="1"
                passwordAttemptWindow="10"
                passwordStrengthRegularExpression="" />
        </providers>
    </membership>

    <profile>
        <providers>
            <add name="AspNetSqlProfileProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Profile.SqlProfileProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </profile>

    <roleManager>
        <providers>
            <add name="AspNetSqlRoleProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Security.SqlRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add name="AspNetWindowsTokenRoleProvider" applicationName="/"
                type="System.Web.Security.WindowsTokenRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </roleManager>
</system.web>
<system.net>
    <connectionManagement>
        <add address="*" maxconnection="999"/>
    </connectionManagement>
</system.net>

Answer 1

Most of the answers provided here address the number of incoming requests to your backend webservice, not the number of outgoing requests you can make from your ASP.net application to your backend service.

It's not your backend webservice that is throttling your request rate here, it is the number of open connections your calling application is willing to establish to the same endpoint (same URL).

You can remove this limitation by adding the following configuration section to your machine.config file:

<configuration>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="65535"/>
    </connectionManagement>
  </system.net>
</configuration>

You could of course pick a more reasonable number if you'd like such as 50 or 100 concurrent connections. But the above will open it right up to max. You can also specify a specific address for the open limit rule above rather than the '*' which indicates all addresses.

MSDN Documentation for System.Net.connectionManagement

Another Great Resource for understanding ConnectManagement in .NET

Hope this solves your problem!

EDIT: Oops, I do see you have the connection management mentioned in your code above. I will leave my above info as it is relevant for future enquirers with the same problem. However, please note there are currently 4 different machine.config files on most up to date servers!

There is .NET Framework v2 running under both 32-bit and 64-bit as well as .NET Framework v4 also running under both 32-bit and 64-bit. Depending on your chosen settings for your application pool you could be using any one of these 4 different machine.config files! Please check all 4 machine.config files typically located here:

• C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config
• C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config

Answer 2

I realize the question might be rather old, but you say the backend is running on the same server. That means on a different port, probably other than the default port 80.

I've read that when you use the "connectionManagement" configuration element, you need to specify the port number if it differs from the default 80.

LINK: maxConnection setting may not work even autoConfig = false in ASP.NET

Secondly, if you choose to use the default configuration (address="*") extended with your own backend specific value, you might consider putting the specific value first! Otherwise, if a request is made, the * matches first and the default of 2 connections is taken. Just like when you use the section in web.config.

LINK: <remove> Element for connectionManagement (Network Settings)

Hope it helps someone.

Answer 3

Might it be possible that you're using a WCF-based web service reference? By default, the ServiceThrottlingBehavior.MaxConcurrentCalls is 16.

You could try updating your service reference behavior's <serviceThrottling> element

<serviceThrottling
    maxConcurrentCalls="999" 
    maxConcurrentSessions="999" 
    maxConcurrentInstances="999" />

(Note that I'd recommend the settings above.) See MSDN for more information how to configure an appropriate <behavior> element.

Answer 4

Have you tried to set the value of the static DefaultConnectionLimit property programmatically?

Here is a good source of information about that true headache... ASP.NET Thread Usage on IIS 7.5, IIS 7.0, and IIS 6.0, with updates for framework 4.0.

Answer 5

See the "Threading" section of this page: http://msdn.microsoft.com/en-us/library/ff647786.aspx, in conjunction with the "Connections" section.

Have you tried upping the maxconnection attribute of your processModel setting?

Answer 6

If it is not defined in the web service or application or server (apache or IIS) that is hosting the web service consumable then you could create infinite connections until failure

Answer 7

while doing performance testing, the measure i go by is RPS, that is how many requests per second can the server serve within acceptable latency.

theoretically one server can only run as many requests concurrently as number of cores on it..

It doesn't look like the problem is ASP.net's threading model, since it can potentially serve thousands of rps. It seems like the problem might be your application. Are you using any synchronization primitives ?

also whats the latency on your web services, are they very quick to respond (within microseconds), if not then you might want to consider asynchronous calls, so you dont end up blocking

If this doesnt yeild something, then you might want to profile your code using visual studio or redgate profiler