Low level keyboard hook issue with an overlay application

Question

First post here, I've been stewing over the correct solution to this issue for years.

I have my own UI engine with its own keyboard handling, and am using it to display a game overlay. The game overlay itself is transparent to both keyboard and window events, in order to be minimally obtrusive towards the game, but in order for the overlay itself to be interactive I needed to resort to keyboard and mouse hooks to block some of the events from reaching the game. For mouse input, this is pretty trivial and works perfectly. It's the low level keyboard hook where I ran into issues.

At this point, I have something that's usable in most cases. I managed to work around several issues involving dead keys and bad input, but never managed to create a hook that could actively block keyboard input to the game - something always goes wrong.

Active blocking of keyboard input would, for example, be most useful when the user is trying to write some text in a textbox on the overlay and doesn't want the game processing the same keystrokes.

My current issue is that, if I block the keyboard inputs by returning a nonzero value in the hook process, the UI engine of the overlay stops perceiving the Ctrl key's state, which results in not being able to copy/paste into the overlay's textboxes. Interestingly, before Alt-Tab'ing, everything works fine, but after that, the Ctrl keypress the hook grabs turns from VK_CONTROL to VK_LCONTROL. And even more interestingly, neither GetKeyState(VK_CONTROL) nor GetAsyncKeyState(VK_CONTROL) nor GetAsyncKeyState(VK_LCONTROL) on the UI side register the Ctrl key as pressed.

The code for the keyboard hook below is a bit of a mess, due to years of experimentation and workarounds. I'll comment it as best as I can.

LRESULT __stdcall KeyboardHook( int code, WPARAM wParam, LPARAM lParam )
{
  // this is an early exit if the game tells me that it actively has focus
  if ( disableHooks || mumbleLink.textBoxHasFocus )
    return CallNextHookEx( 0, code, wParam, lParam );

  // the following two early exits are remnants from earlier experimentation
  if ( code < 0 )
    return CallNextHookEx( 0, code, wParam, lParam );

  if ( wParam != WM_KEYDOWN && wParam != WM_KEYUP && wParam != WM_CHAR && wParam != WM_DEADCHAR && wParam != WM_UNICHAR )
    return CallNextHookEx( 0, code, wParam, lParam );

  // this checks if either the game or the overlay are in focus and otherwise ignores keyboard input
  auto wnd = GetForegroundWindow();
  if ( code != HC_ACTION || !lParam || ( wnd != gw2Window && App && wnd != (HWND)App->GetHandle() ) )
    return CallNextHookEx( 0, code, wParam, lParam );

  // this ignores the overlay itself if it's in focus for some odd reason
  if ( App && wnd == (HWND)App->GetHandle() )
    return CallNextHookEx( 0, code, wParam, lParam );

  KBDLLHOOKSTRUCT *kbdat = (KBDLLHOOKSTRUCT*)lParam;
  UINT mapped = MapVirtualKey( kbdat->vkCode, MAPVK_VK_TO_CHAR );

  // this bool tests if the overlay has a textbox in focus and the keyboard input should be blocked from propagating further
  bool inFocus = App->GetFocusItem() && App->GetFocusItem()->InstanceOf( "textbox" );

  // forcefully inject a WM_CHAR message to the overlay's UI engine - never figured out how to trigger a message that would be translated into a WM_CHAR properly
  if ( !( mapped & ( 1 << 31 ) ) && !inFocus && wParam == WM_KEYDOWN )
    App->InjectMessage( WM_CHAR, mapped, 0 );

  if ( inFocus )
  {
    PostMessage( (HWND)App->GetHandle(), wParam, kbdat->vkCode, 1 | ( kbdat->scanCode << 16 ) + ( kbdat->flags << 24 ) );

    /////////////////////////////////////////////////
    return 1; // this is where the key input should be blocked, but it causes the mentioned issues with the ctrl key (and probably others too)
    /////////////////////////////////////////////////
  }

  return CallNextHookEx( 0, code, wParam, lParam );
}

The UI engine itself checks the Ctrl, Shift and Alt states through GetKeyState() because tracking these through WM_SYSKEYDOWN messages would, for example, result in an Alt-Tab having the Alt key stuck since the window would never receive the WM_SYSKEYUP message. The function that checks the state of the Ctrl/Shift/Alt keys is called on several different WM_... messages when necessary. However, as soon as the VK_LCONTROL messages start being intercepted by the keyboard hook instead of the VK_CONTROL ones, this function always reports all keys as being unpressed.

Why don't you just let your text boxes have input focus as normal? Then you wouldn't need any of this stuff. — Jonathan Potter
The UI engine in question is a directx drawn custom application framework, not a standard windows form. All widgets in the system are custom, including the textbox. As such there is no thing as “input focus as normal”. Also since the overlay itself is transparent to keyboard and mouse input (hence the need for the hooks) even in the case you mention the problem would still be the same: i’d need to use a keyboard hook to feed the overlay with window messages and keep some of the inputs from reaching the game. — BoyC
Without normal focus, how do you distinguish keyboard input from textbox and other widgets? — Strive Sun
What you're talking about requires the window be non-input-transparent to catch window messages directly, but that goes against the whole idea of a full-screen overlay that displays information (and sometimes being interactive) without interfering with what's happening behind it. And yes, the UI does have focus handling code, you can see in the posted how that is used to filter the window hook's messages. — BoyC

Perneky László Perneky László · Accepted Answer · 2020-06-16T06:17:29

You could try a different approach. If your overlay is the active window during this, then you can handle keyboard and mouse events without the hook, and if you want to forward an event to the game, you can just synthetize the event for the game window.

Low level keyboard hook issue with an overlay application

1 Answers