High performance C# TCP server problem: No connection could be made because the target machine actively refused it

6
votes

I have developed a TCP server according to your advises: High performance TCP server in C#

It is based on asynchron pattern.

I also developed a stress test application to test its performance. My server can get thousands of connections paralelly from my stress test app, can parse data and save it to my database.

When I stress my server, I can get "System.Net.Sockets.SocketException "No connection could be made because the target machine actively refused it" error from my server, so I have to reconnect to it. If I test it with 5000 concurrent connections, I have to try connect again because of this problem 10-20% of the connections, if I test it with 10K concurrent connections, it can be 30-40%. Sometimes it can be - very rarely - more, than 50%. It seems it can not handle connection accepts: I make new connections from my stress test as heavily as my test machine can - about 120 connections/sec.

So, what can cause this kind of exception? How to handle it? What to do in server side implementation to avoid this problem? How to tune TCP connection accept?

Thanks in advance!

2
c#exceptiontcpconnection
I think this is the whole point of stress testing = to make sure your machines can handle the load. And this is how you find out what happens when you get near the limit of your capacity. – DOK
Yes, but I'd like to make it better - if possible... – Tom
If you want to make your software better make sure it handles the exceptions correctly. You're no going to prevent/avoid them. – Henk Holterman
Thx :)) The main reason I asked my questions is I don't know exactly that I reach the maximum performance of my machine or I can make it better by improving some settings or may be my code etc... – Tom
Do you have a can somewhere published a piece of server code? – dariol

2 Answers

7
votes

You might be running out of available ports every now and then. You can view this easily using SysInternals' TcpView utility.

On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.

There are two registry entries that can reduce this time interval: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay

and increase the max number of ports that can be opened by an app: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort

EDIT: MaxFreeTcbs seems to be a third setting which could help (I haven't tried this yet), mentioned in this TechNet article which has more advice on tracking down odd network problems. HTH.

2
votes

You are making connections faster than the software can listen for new connections, or in other words you are reaching the connections per second limit of that port. I think you can double the amount of connections per second by listening to a second port, client side you should just reconnect when you get the exception.

There are also limits applied to the amount of connection, for these see Chris O's answer.