I'm using and improving on an open source MFC work-alike library called FFC. Sometimes the library associates the wrong window handle to a dialog object, which means the C++ object can't be found later when the correct handle is looked up. In particular, this is happening when the application opens its root window, which is a dialog that it opens with a call to DoModal.
In its DoModal function, the FFC library uses a... "surprising" way to attach the handle to the dialog object. It stashes the "this" pointer in a global variable and hooks a function to be called on all window messages before calling the DialogBox function. This hook function it registered in term assumes the handle from the first message it receives is the handle for the window in the global variable, and attaches that handle to it.
Sometimes, this works. Often - and I don't know if it's because of intrusive things done by the McAfee scanner on my work computer, or because my program starts from a console window, or something else - many unrelated messages will be captured before a message actually meant for the modal dialog comes through.
At first I thought it was because FFC wasn't making sure the message it looks for is "WM_CREATE". I added this check, but it didn't fix the problem. Turns out one or more of the spurious messages are also WM_CREATE messages! Before it gets the one for the real dialog, the first WM_CREATE it receives is a handle for a window with blank window text and rectangle 0,0-0,0.
So is this really the correct or canonical way to get the handle for a modal dialog? It seems unreliable. (Note that because the dialog is modal, you can't use the return value from CreateWindowEx because the DialogBox function doesn't return until the modal dialog is closed.) Is this really how MFC does it? Is there a better way? Could I associate some data with the dialog or look for data that should be associated with it to make sure I have the right window handle? (For instance checking the template parameter passed to the dialog box call, if I can get that back from the handle somehow.)
CreateWindowEx(or one of its friends), lets the hook callback set up the instance pointer, and unhooks the current thread. All this allows for the OS to re-order messages as it sees fit, and MFC will still have an instance pointer properly attached to a window handle, when the first message arrives, whatever that may be on any given version of Windows. If you see spurious message get in the way, you are probably violating the thread-affinity rules: GUIs need to be single-threaded. – IInspectable