How to install a hook on a process created with CreateProcess?

Question

Here is what I tried :

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow) {
  cout << "Starting Notepad++..." << endl;
  STARTUPINFO startupInfo;
  PROCESS_INFORMATION processInformation;

  // set the size of the structures
  ZeroMemory(&startupInfo, sizeof(startupInfo));
  startupInfo.cb = sizeof(startupInfo);
  ZeroMemory(&processInformation, sizeof(processInformation));

  char commandLine[] = "C:\\Program Files\\Notepad++\\Notepad++.exe";

  // start the program up
  BOOL res = CreateProcess(NULL,   // the path
    commandLine,        // Command line
    NULL,           // Process handle not inheritable
    NULL,           // Thread handle not inheritable
    FALSE,          // Set handle inheritance to FALSE
    0,              // No creation flags
    NULL,           // Use parent's environment block
    NULL,           // Use parent's starting directory
    &startupInfo,            // Pointer to STARTUPINFO structure
    &processInformation             // Pointer to PROCESS_INFORMATION structure (removed extra parentheses)
    );

  if (res) {
    if (!(mouseHook = SetWindowsHookEx(WH_MOUSE_LL, MouseHookCallback, NULL, processInformation.dwThreadId))) {
        cout << "Failed to install mouse hook :" << endl << getLastErrorAsString() << endl;
    }

    WaitForSingleObject( processInformation.hProcess, INFINITE );
    CloseHandle( processInformation.hProcess );
    CloseHandle( processInformation.hThread );
  } else {
    cout << "Failed to start Notepad++" << endl;
  }
  return 0;
}

It starts Notepad++ successfully, but it fails to install the hook and GetLastError return the following error : The parameter is incorrect.. I have no idea which parameter is incorrect. However, the program finishes normally when I close Notepad++.

Since I start the process in the main program and the hook callback is also in the main program, I should be able to install a hook without doing any dll injection.

I haven't touched to c++ in years, and I've never been into system development, so I may be wrong in my way to do it, so can you explain to me where my error is ?

EDIT : You're all telling me that I need to inject a dll to hook a specific process, but this is from the windows documentation of SetWindowsHookEx about the hMod parameter (3rd parameter):

A handle to the DLL containing the hook procedure pointed to by the lpfn parameter. The hMod parameter must be set to NULL if the dwThreadId parameter specifies a thread created by the current process and if the hook procedure is within the code associated with the current process.

My Thread has been created by the current process and my hook procedure is inside the code of my current process, so why it doesn't work when I'm using a not low-level hook (WH_MOUSE)?

To find out about incorrect arguments, check out the documentation. But as I recall, to inject a hook in another process you need to have a DLL. — Cheers and hth. - Alf
WH_MOUSE_LL is global hook. it can not be set to concrete thread. this is system wide — RbMm
@RbMm my bad, I missed that in the documentation, any way to install a mouse hook for a specific process only? — François MENTEC
and if you set say WH_MOUSE hook - when you use thread id for another process - you need provide DLL - which will be injected to this process, because this hook called in process context. so hMod must be not 0 and must be same bitness (32 or 64) as target process — RbMm
"My Thread has been created by the current process" - That's not correct. processInformation.dwThreadId has been created by the process created by your CreateProcess call. That doesn't change anything, though: You do not need to provide a DLL or inject any code into any process. As clearly documented: "This hook is called in the context of the thread that installed it." — IInspectable

IInspectable IInspectable · Accepted Answer · 2018-07-27T09:41:27

Low-level hooks are executed, before the destination of the input has even been evaluated. That's the reason, why low-level hooks need to be global, as explained in the documentation for SetWindowsHookEx. You cannot pass a non-zero value for the dwThreadId parameter.

How to install a hook on a process created with CreateProcess?

1 Answers