WSAsend to all connected socket in multithreaded iocp server

Question

I work on IOCP Server (Overlapped I/O , 4 threads, CreateIoCompletionPort, GetQueuedCompletionStatus, WSASend etc). I also created an auto-reset event and put the handle in the OVERLAPPED structure for each asynchronous I/O operations.

And the question is: How to send buffer too all connected sockets properly? Each socket is stored in linked list of context info structures.

I'm not sure is that approach below is ok?

...
DWORD WINAPI WorkerThread() { // 1 of 4 workthread
...
GetQueuedCompletionStatus(...);
...
PPER_SOCKET_CONTEXT  pTmp1, pTmp2;
pTmp1 = g_pCtxtList; // start of linked list with sockets
EnterCriticalSection(&g_CriticalSection);
while( pTmp1 ) 
{  
pTmp2 = pTmp1->pCtxtBack;
         WaitForSingleObject(pTmp1->pIOContext->Overlapped.hEvent,infinite);                
          // if there is any pending wsasend on this socket,wait to completed, so
          // we can post another wsasend using the same overlapped structure
          // and buffer
           WSASend(pTmp1->Socket,...,&(pTmp1->pIOContext->Overlapped), NULL);
       pTmp1 = pTmp2;
 }
LeaveCriticalSection(&g_CriticalSection);
...
}

And what happens if another thread also tries to do the same work at same time?
Is that good idea to use GQCS and waits function in all threads?
Any clue about wsasends to all clients in multithreaded iocp server will be appreciated.
thx

@Len Holgate - why did you delete your answer? It's better than mine, (because I did not read the OP question properly). I have added that the refCount needs to be decremented atomically, but basically, your answer should work. Put it back!! — Martin James
Please upvote Len Holgate answer when/if he undeletes it - it's better than mine. — Martin James
Martin, sorry, I deleted because I didn't think I'd read the OP question (based on my reading it again in the context of your answer :) ) — Len Holgate

Len Holgate Len Holgate · Accepted Answer · 2013-11-04T09:15:04

As Martin says, this will perform terribly and likely kill the performance of anything that uses the list of sockets that you lock for the entire duration of your send to all connections. You don't say if this is UDP or TCP but if it's TCP be aware that you are now handing control of your server's performance over to the clients as TCP flow control on a slow client connection may cause the write completion to be delayed (see here)- and I assume you're using the write completion to trigger the event?

I assume that your actual requirement is that you want to avoid copying the data on the server and allocating multiple buffers, one for each connection either due to memory constraints or because you've profiled the memory copy and found that it's expensive.

The way I deal with this is to have a single reference counted buffer and a 'buffer handle' which is just a slightly extended overlapped structure which references your single data buffer and provides the WSABUF that you need. You can then issue a 'fire and forget' write to each connection using a unique 'buffer handle' all of which refer to the single underlying buffer. Once all writes complete the ref count on the buffer is reduced to zero and it cleans up - and as Martin says, that clean up is best achieved by putting the buffer into a pool for later reuse.

Note: I'm not sure that I actually understand what you are trying to do (so I orginally deleted my answer), let me know if I'm not following and I'll adjust...

WSAsend to all connected socket in multithreaded iocp server

2 Answers