When a Windows Hook is Called, Which Code is Run?

Question

The documentation for SetWindowsHookEx says,

If an application requires the use of hooks in other processes, it is required that a 32-bit application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit processes, and a 64-bit application call SetWindowsHookEx to inject a 64-bit DLL into 64-bit processes.

This strongly implies that when the hook filter callback is called, it is the code inside the hooked application's copy of the hook DLL that is run.

However, when I install the hook, I need to pass a function pointer to my filter callback function. This strongly implies that it is the code inside the application that installs the hook that is run when the hook is called. This is because a function pointer inside one process' address space is not valid inside the address space of another process.

So which is it? When a hook filter callback is called, which code is actually run, the code inside the targeted (hooked) application, or the code inside the application that installed the hook?

Thank you

Answer 1

The SetWindowsHookEx doesn’t take just a function pointer: it takes a function pointer together with a handle of the DLL the function is defined in. Given that, the kernel can calculate function address relative to DLL base address. Then it (in some cases at least) injects the DLL in other processes on the system, recalculates address (if necessary; DLL base address is usually fixed so the code doesn’t need to be position-independent. Relocation is still possible, though) and setups the hook in the target process. (According to Hans Passant’s comment, that only happens for some hooks, while others run in the source process).

That means that you can pass address of any function existing in a DLL; it doesn’t need to be exported AFAIK (that seems to be the reason not to use names instead) but it needs to exist in its on-disk image.