WSARecv sometimes return “invalid handle (error no 6)” for a socket associated with an IOCP port. (C++)

Question

I'm trying to write a server which can support many clients connections simultaneously so I'm trying to do it with IOCP. So let me brief about my code flow and then I can explain my problem. First of all, server is opening a port for listening and waiting on an "accept" call for new incoming connections. For reference I have used same code as mentioned here So it accepts every new incoming connection and returns a new socket descriptor (sd), and then it marks as nonblocking with:

arg = 1;
ioctlsocket(sd, FIONBIO, &arg);

and then enable TCP_NODELAY:

level = IPPROTO_TCP;
optName = TCP_NODELAY;
value = 1;
setsockopt(sd, level, optName, (const char*)&value, sizeof(value));

thereafter associating with an IOCP port as:

CreateIoCompletionPort((HANDLE)sd, iocp_port, (DWORD)completion_key, 4);

completion_key is a class object which is nothing but a container, it contains data buffer, overlapped-buffer, query-type recv/send etc. and in last issuing a read call:

WSARecv(sd, wsabuf, 1, &bytes, &flags, overlapped, NULL);

wsabuf and overlapped are part of completion_key object.

In 90% cases it works fine i.e. when there is some incoming data available on this socket "GetQueuedCompletionStatus" gets unblocked and it has valid data. But sometimes WSARecv call returns with an error and GetLastError() returns 6 which is "invalid handle" error. I'm bit bewildered why it's happening so.

The way I'm creating an iocp port:

iocp_port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);

and there are threads which are waiting on "GetQueuedCompletionStatus".

Where is the overlapped block? You need a different one for each WSARecv call, both in the first one you issue in the accept() loop and those that you, (presumably), issue in the handler threads after handling a completion message. — Martin James
It's common in OO to have the OVL struct as a member of the buffer class, so as to ensure that each overlapped I/O call has its own OVL block. — Martin James
yes, I know that. completion_key is a class object which contains an overlapped buffer. So for each WSARecv call a different overlapped buffer is passed, I'm updating above content with same. — nik_kgp
I'm pretty sure you know what you're doing (you kinda have to if you're using IOCP), but I'll mention it regardless. Forget everything you know about anything once you reach your GetQueueCompletionStatus() loop. All too often I see people go through all the setup, fire up their GQCS loop, then utterly forget that when that function returns with success the only thing you can rely on is the LPOVERLAPPED returned with it. Having read what I just wrote, i realize if someone has never used IOCP its going to be greek to them. But if you have, you will know exactly what I'm referring to. — WhozCraig
@WhozCraig - heh, yes, I use only that. I load a buffer+OVL instance pointer into the hEvent field and cast it back in the handler thread. Each buffer has a pointer to its socket object so I can issue another WSARecv. Not sure what I would do on 64-bit:( — Martin James

nik_kgp nik_kgp · Accepted Answer · 2013-12-16T14:47:52

I monitored all system calls which were happening in background. WSARecv internally calls NtDeviceIoControlFile and there is an argument "Event" which is same as what is passed in lpOverlapped structure of WSARecv as hEvent. I wasn't setting hEvent to NULL, so it was taking some garbage value, when it was NULL then NtDeviceIoControlFile returned successfully and for other cases it returned "INVALID_HANDLE" error. Unfortunately, it was NULL most of the time.

WSARecv sometimes return “invalid handle (error no 6)” for a socket associated with an IOCP port. (C++)

1 Answers