How to trace overhead added by Application Request Routing?

9
votes

We have a Delphi SOAP service which needs to be SSL-enabled. I opted to use an IIS ARR reverse proxy to do SSL offloading for ease of configuration (compared to OpenSSL and manual certificate + passphrase management). ARR works, but it adds an insane amount of overhead... Response time went from under 2 seconds to 19 seconds for 18 service requests (about 60Kb compressed total).

I added timestamp logging to client & server for when messages are both sent and received. It shows about 1 second added to each request routing through ARR between sending from the client and receipt by the service. The response is routed back very quickly, only the request routing via ARR is slow (see image below).

How can I trace the source of overhead? Is ARR not suited to this use case? I tried tweaking and disabling most settings, including caching. I tried different hosts with clean IIS setups, including a production Windows Server 2012. SSL itself is not the overhead, just having an ARR HTTP reverse proxy causes the delay.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:8987/{R:1}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Request & response samples from Fiddler:

Client and server timestamps grid

I0IS Tracing shows overhead from ARR

4
iissoapiis-7arriis-arr
Shouldn't see this type of delay. First thing that comes to mind is your application pool timeout setting, are they being shutdown and the delay is the pool starting up on each request? Make sure that they never shutdown due to idle. – Brock Hensley
@dirt thanks for replying. My application pool idle shutdown is set to two days, and recycle only on memory usage limit breaches. I am currently on "No Managed Code" and integrated pipeline. IIS tracing shows the delay is on ARR's EXECUTE_REQUEST_HANDLER (I added the image after your comment). – carlmon
@carlmon Just wondering if you ever solved this? I am suspect about the ARR performance in my environment (although it's nothing this bad), ps: I see ARR 3.0 was just released out of beta. – CameraSchoolDropout
No solution. This is all ARR 3.0. We released without SSL and will probably use OpenSSL in Delphi in our next update for SSL support. – carlmon
Changing localhost to 127.0.0.1 in rewrite Action URL reduced the load time to a significant amount. – Raghav

4 Answers

10
votes

We have the same problem. I found the root, it's in System.Net.Sockets.Socket.DoConnect The problem is related to IPv6:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/203b6230-e4c0-477c-9a0a-0c21a7ad1615/strange-onesecond-delay-with-tcpconnections-to-localhost?forum=clr

http://msdn.microsoft.com/en-us/library/115ytk56.aspx

"If IPv6 is enabled and the TcpClient(String, Int32) method is called to connect to a host that resolves to both IPv6 and IPv4 addresses, the connection to the IPv6 address will be attempted first before the IPv4 address. This may have the effect of delaying the time to establish the connection if the host is not listening on the IPv6 address."

To resolve it for loopback requests you need to disable IPv6 on a machine, see p.4-5-6: https://stackoverflow.com/a/12403731

4
votes

My advice: use IIS for the apps, use Apache HTTP Daemon for the proxying.

In the past I've used various pieces of software and hardware for SSL offloading (starting somewhere in 2003 I think). Each with their own pricing level and features. The last years I've switched to solely using Apache HTTP Daemon for that purpose. Even in combination with IIS and on Windows. Apache is easy to configure once you have a running sample and more easily grows to more complex scenarios with forwarding and renaming.

Some instructions how to use Apache HTTP Daemon on Windows as SSL offloading engine can be found on http://www.invantive.com/about-invantive/news/entryid/897/ssl-offloading-for-apache-tomcat.

1
votes

Turning off IPv6, as Dmitry suggested, solved this for me.

You could also use 127.0.0.1 in your rewrite, instead of localhost, to force IPv4.

0
votes

I would say you have something screwy in your setup. We currently run ARR for SSL offloading and conducted throughput testing at significant volume and ARR showed little to no affect on throughput.

I would agree with Brock and say check your App Pool settings first. Basically an ARR App Pool should be set to never recycle.

I would recommend watching episodes 32-38 starting at: http://dotnetslackers.com/articles/iis/Bindings-and-Rules-for-Application-Request-Routing-ARR-Week-32.aspx