How close a socket (with IOCP) properly after sending?

Question

I'm having problem with Winsock2 using IOCP (overlapped IO mode) when I need to close the connection after sending the requested data.

I've discovered if I send some data and close the socket immediately after sending, then the data will not be (or partially will be) sent, because there is no time to send out the packet before closing.

I've also discovered there are multiple disconnection methods, like WSA_SendDisconnect(), which I use for now.

Now the current implementation is something like this:

Sending out the data
Setting a flag that represents the intention to close the connection after sending
When IOCP event happens regarding the send event if the flag is set and bytecount > 0, then I send again an empty buffer, just to make sure all data has been sent
When IOCP event happens regarding the send event if the flag is set and bytecount == 0 then I clear the flag and call WSA_SendDisconnect() with empty buffer, to send disconnect message
When IOCP event happens regarding the send event without the flag set and bytecount == 0 then I call closesocket() and destroy the context.

With this procedure I could make it almost sure the socket will disconnect only after all data has been sent, actually it works very well, but when I stress test the program in some very rare cases 10-15 times out of 1000000 tests something is wrong. (It would be very hard to determine the exact problem, the program works as a web server, and I'm stresstesting it with apachebench and siege, after the test I get a summary where I see sometimes 10-15 failed requests.)

I was pretty sure the failure is because the socket can not send the entire packet before closing, so I've placed a little delay before socketclose() and since that I get zero failed requests, but of course, this is not a solution, just a way to filter out what may cause the problem.

The method I've implemented obviously lacks a proper way to detect if the socket has sent everything before closing, even though I start closing only after the last dummy write has finished.

What would be the best solution to close the socket only after all sending buffer has actually be written to the network?

Ps: I've tried playing with NoDelay and setting LingerState, they did not help.

XY problem. TCP sockets are completely flushed of data before the FIN is sent. Your question is based on a false premiss. And sending an empty buffer over TCP does exactly nothing. — user207421
It seems using Winsock with IOCP calling socketclose() immediately after writing send buffer closes the socket before flushing. I know very well sending empty packet not just over TCP but over any Ethernet layer does nothing. I just want to ensure the packet sequence ends with a 0 bytes long packet, that returns a successfull write event immediately without having any effect on the actual traffic, so that I can close the socket. It is not more than just a marker in my program flow. — beatcoder
Actually sending an empty UDP datagram does work. Anyway all you should have to do is not close until you get the I/O completion for the last write. This shouldn't be hard to arrange. — user207421
call DisconnectEx with TF_REUSE_SOCKET. when this request complete you can close socket — RbMm
@beatcoder "I have not found any practical solution to detect if the data has been flushed from the buffer" - because no such signal exists. If you really needed this, you would have to monitor the network hardware directly, such as with libpcap, to see when the data arrives on the hardware. — Remy Lebeau

beatcoder beatcoder · Accepted Answer · 2020-08-23T22:34:36

Ok, I can surely say this problem is solved after a lot of testing, this solution is working fine in my webserver for months without any issue in every circumstances. The anwers were helpful, but the final solution is the following:

        try
        {
            socket.Shutdown(SocketShutdown.Both);
        }
        catch (SocketException ex)
        {
            // handle exception
        }
        finally
        {
            socket?.Close();
        }

The "trick" is to initiate a disconnect with SocketShutdown.Both and finally close the socket if it still exists. I've noticed sometimes (I really don't know why) the shutdown method automatically disposed the socket, so the socket.Close() threw a NullReferenceException, but sometimes not. I worked this issue around with a ? after the socket object referece, so if it is null, then the Close() method just will not be called.

It works fine on Windows and also in Linux (debian) in dotnet core.

Why only this works? Don't know, but with this I'm not having failed requests, packet loss, or improperly closed connections. Just works.

How close a socket (with IOCP) properly after sending?

1 Answers