SetWindowsHookEx() doesn't return an error when hooking 64 bit processes from a 32 bit process?

1
votes

My application is compiled as 32-bit, and since I run on 64-bit Windows 7, my target(notepad.exe) is 64-bit. When I call SetWindowsHookEx() on the first thread that I find of notepad.exe, the DLL doesn't get injected at all, but there is no error returned. I know it's not being injected because on DLL_PROCESS_ATTACH I display message box with the message Attached, and for DLL_PROCESS_DETACH I display a Detached message in a message box. These messages are only displayed once for when I call LoadLibrary() and another time for when my application exits.

According to the MSDN documentation here:

Because hooks run in the context of an application, they must match the "bitness" of the application. If a 32-bit application installs a global hook on 64-bit Windows, the 32-bit hook is injected into each 32-bit process (the usual security boundaries apply). In a 64-bit process, the threads are still marked as "hooked." However, because a 32-bit application must run the hook code, the system executes the hook in the hooking app's context; specifically, on the thread that called SetWindowsHookEx. This means that the hooking application must continue to pump messages or it might block the normal functioning of the 64-bit processes.

Does this mean that it's hooking my own process successfully instead of actually returning an error?

Edit : My hook is of WH_CBT type.

2
c++winapidllhook
This question critically depends on the type of hook - David Heffernan
There is no obligation for it to return an error, it certainly injected some processes. Just not the one you want. Such are the trappings on a 64-bit operating system, you'll need to start a 64-bit version of your program with a 64-bit DLL to also inject the 64-bit processes. - Hans Passant

2 Answers

1
votes

You need to read further the docs:

For a specified hook type, thread hooks are called first, then global hooks. Be aware that the WH_MOUSE, WH_KEYBOARD, WH_JOURNAL*, WH_SHELL, and low-level hooks can be called on the thread that installed the hook rather than the thread processing the hook. For these hooks, it is possible that both the 32-bit and 64-bit hooks will be called if a 32-bit hook is ahead of a 64-bit hook in the hook chain.

In short it depends on the hook type if your dll is injected to the target process at all. If you only want to spy on keyboard and mouse events there is no need to inject yourself into other processes. Windows will call back your hook in your own process.

I suspect your hook type is one of these:

  • WH_MOUSE_LL
  • WH_KEYBOARD_LL

which do not cause any library injection into the target process.

1
votes

What that part of the documentation does not state clearly is that when SetWindowsHookEx fails to inject your 32-bit DLL into a 64-bit process, instead of returning an error, it resorts to using the message loop of the thread that called SetWindowsHookEx to execute the hook procedure, just like how it works with low level mouse/keyboard hooks (WH_MOUSE_LL/WH_KEYBOARD_LL).