How to pass user-defined data to a worker thread using IOCP?

Question

Hey... I created a small test server using I/O completion ports and winsock. I can successfully connect and associate a socket handle with the completion port. But I don´t know how to pass user-defined data-structures into the wroker thread...

What I´ve tried so far was passing a user-structure as (ULONG_PTR)&structure as the Completion Key in the association-call of CreateIoCompletionPort() But that did not work.

Now I tried defining my own OVERLAPPED-structure and using CONTAINING_RECORD() as described here http://msdn.microsoft.com/en-us/magazine/cc302334.aspx and http://msdn.microsoft.com/en-us/magazine/bb985148.aspx. But that does not work, too. (I get freaky values for the contents of pHelper)

So my Question is: How can I pass data to the worker thread using WSARecv(), GetQueuedCompletionStatus() and the Completion packet or the OVERLAPPED-strucutre?

EDIT: How can I successfully transmit "per-connection-data"?... It seems like I got the art of doing it (like explained in the two links above) wrong.

Here goes my code: (Yes, its ugly and its only TEST-code)

struct helper
    {
        SOCKET m_sock;
        unsigned int m_key;
        OVERLAPPED over;
    };


///////

SOCKET newSock = INVALID_SOCKET;
    WSABUF wsabuffer;
    char cbuf[250];
    wsabuffer.buf = cbuf;
    wsabuffer.len = 250;
    DWORD flags, bytesrecvd;


    while(true)
    {
        newSock = accept(AcceptorSock, NULL, NULL);
        if(newSock == INVALID_SOCKET)
            ErrorAbort("could not accept a connection");

        //associate socket with the CP
        if(CreateIoCompletionPort((HANDLE)newSock, hCompletionPort, 3,0) != hCompletionPort)
            ErrorAbort("Wrong port associated with the connection");
        else
            cout << "New Connection made and associated\n";

        helper* pHelper = new helper;
        pHelper->m_key = 3;
        pHelper->m_sock = newSock;
        memset(&(pHelper->over), 0, sizeof(OVERLAPPED));
        flags = 0;
        bytesrecvd = 0;

        if(WSARecv(newSock, &wsabuffer, 1, NULL, &flags, (OVERLAPPED*)pHelper, NULL) != 0)
        {
            if(WSAGetLastError() != WSA_IO_PENDING)
                ErrorAbort("WSARecv didnt work");
        }
    }

    //Cleanup
    CloseHandle(hCompletionPort);
    cin.get();
    return 0;
}

DWORD WINAPI ThreadProc(HANDLE h)
{
    DWORD dwNumberOfBytes = 0;
    OVERLAPPED* pOver = nullptr;
    helper* pHelper = nullptr;
    WSABUF RecvBuf;
    char cBuffer[250];
    RecvBuf.buf = cBuffer;
    RecvBuf.len = 250;
    DWORD dwRecvBytes = 0;
    DWORD dwFlags = 0;
    ULONG_PTR Key = 0;

    GetQueuedCompletionStatus(h, &dwNumberOfBytes, &Key, &pOver, INFINITE);

    //Extract helper
    pHelper = (helper*)CONTAINING_RECORD(pOver, helper, over);


    cout << "Received Overlapped item" << endl;
    if(WSARecv(pHelper->m_sock, &RecvBuf, 1, &dwRecvBytes, &dwFlags, pOver, NULL) != 0)
        cout << "Could not receive data\n";
    else
        cout << "Data Received: " << RecvBuf.buf << endl;

    ExitThread(0);
}

Chris Becke Chris Becke · Accepted Answer · 2010-12-03T10:24:40

If you pass your struct like this it should work just fine:

helper* pHelper = new helper;
CreateIoCompletionPort((HANDLE)newSock, hCompletionPort, (ULONG_PTR)pHelper,0);
...


helper* pHelper=NULL;
GetQueuedCompletionStatus(h, &dwNumberOfBytes, (PULONG_PTR)&pHelper, &pOver, INFINITE);

Edit to add per IO data:

One of the frequently abused features of the asynchronous apis is they don't copy the OVERLAPPED struct, they simply use the provided one - hence the overlapped struct returned from GetQueuedCompletionStatus points to the originally provided struct. So:

struct helper {
  OVERLAPPED m_over;
  SOCKET     m_socket;
  UINT       m_key;
};

if(WSARecv(newSock, &wsabuffer, 1, NULL, &flags, &pHelper->m_over, NULL) != 0)

Notice that, again, in your original sample, you were getting your casting wrong. (OVERLAPPED*)pHelper was passing a pointer to the START of the helper struct, but the OVERLAPPED part was declared last. I changed it to pass the address of the actual overlapped part, which means that the code compiles without a cast, which lets us know we are doing the correct thing. I also moved the overlapped struct to be the first member of the struct.

To catch the data on the other side:

OVERLAPPED* pOver;
ULONG_PTR key;
if(GetQueuedCompletionStatus(h,&dw,&key,&pOver,INFINITE))
{
  // c cast
  helper* pConnData = (helper*)pOver;

On this side it is particularly important that the overlapped struct is the first member of the helper struct, as that makes it easy to cast back from the OVERLAPPED* the api gives us, and the helper* we actually want.

How to pass user-defined data to a worker thread using IOCP?

3 Answers