Several programs crash when unhooking with UnhookWindowsHookEx()

Question

I am doing a global hook to add my DLL to the hook chain:

HHOOK handle = SetWindowsHookEx(WH_CALLWNDPROC, addr, dll, 0);

Inside my DLL I am using Detours to intercept several WINAPI function calls. Everything works fine, except for WaitForSingleObject calls. Whenever I add WaitForSingleObject to the detoured functions, several programs crash when I unhook my DLL (Chrome, Skype, ...). Here is how the DLL looks:

DWORD (WINAPI* Real_WaitForSingleObject)( HANDLE hHandle, DWORD dwMilliseconds) = WaitForSingleObject;
DWORD WINAPI Mine_WaitForSingleObject(HANDLE hHandle, DWORD dwMilliseconds);
INT APIENTRY DllMain(HMODULE hDLL, DWORD Reason, LPVOID Reserved) {

    switch(Reason) {
        case DLL_PROCESS_ATTACH: 
            DetourTransactionBegin();
            DetourUpdateThread(GetCurrentThread());
            DetourAttach(&(PVOID&)Real_WaitForSingleObject, Mine_WaitForSingleObject);
            DetourTransactionCommit();
            break;
        case DLL_PROCESS_DETACH: 
            DetourTransactionBegin(); 
            DetourUpdateThread(GetCurrentThread());
            DetourDetach(&(PVOID&)Real_WaitForSingleObject, Mine_WaitForSingleObject);
            DetourTransactionCommit();
            break;
        case DLL_THREAD_ATTACH:

            break;
        case DLL_THREAD_DETACH:

            break;
    }
    return TRUE;
}
DWORD WINAPI Mine_WaitForSingleObject(HANDLE hHandle, DWORD dwMilliseconds) {

    return Real_WaitForSingleObject(hHandle, dwMilliseconds);
}

extern "C" __declspec(dllexport) int meconnect(int code, WPARAM wParam, LPARAM lParam) {

    return CallNextHookEx(NULL, code, wParam, lParam);
}

Could someone help me to understand why this is happening and how I can get around that Problem? Thanks!

Hardly surprising that such egregious hackery would lead to problems. — David Heffernan
Presumably, when you remove the hook the DLL gets unloaded. You remove the detour, but that won't stop threads that are already in the middle of a detoured call from crashing. The only reason WaitForSingleObject is particularly likely to cause a crash is that it tends to block for long periods. (The other detoured functions will only crash occasionally, because most of the time they won't be running at the exact point at which the DLL is unloaded.) — Harry Johnston
Is there a way to safely unload the DLL without causing detoured calls to crash? Isn't DetourDetach() supposed to prevent such crashes? — moccajoghurt
I don't see how it could, particularly in this context. But you should consult the documentation. — Harry Johnston
What you are doing is beyond the pale. You should not expect to be able to do this. How could you expect code to continue executing after you unload it? How could it be possible to unload the code safely? — David Heffernan

Soonts Soonts · Accepted Answer · 2015-11-15T15:06:54

I think this is happening because many programs (Chrome, Skype, ...) have a thread pool, where background thread[s] are waiting on WaitForSingleObject() for something interesting for them to happen, and when it does happen, that thread[s] wake up and do something.

So, your thread A is calling DetourDetach while another thread B of the same process is currently inside Mine_WaitForSingleObject() Then DLL unloads, and everything crashes. You can verify by using debugger, attach to that problematic process, set breakpoint in DLL_PROCESS_DETACH, and when the breakpoint will hit, look through the stacks of another threads for Mine_WaitForSingleObject.

I’m not sure how to fix. But, one way that you might try — enumerate threads, and call DetourUpdateThread() for every thread of the process. This way, maybe the Detours will do something about that.

Several programs crash when unhooking with UnhookWindowsHookEx()

2 Answers