MFC modal dialog close error

Question

I have a strange error and spend hours in the debugger without finding a solution. (But it helped me to fixed another error that you should never call EndDialog from a WM_KICKIDLE task).

My problem is that i have a main window and a modeless dialog window wich raises a modal subdialog window. When the subdialog window is closed. The modeless dialog window turns itself into a modal window. My code really does leave the modal loop. And if i close the now modal window it behaves like an invisble modal window is active, meaning no interaction is possible anymore.

When i only run a modal dialog on top of the main window it is closed fine.

BTW: The main window is not the one available view CWinApp::m_pMainWnd but a new create FrameWindow. I hide the p_MainWnd and use it as an invisible message only window. From some comments and my debugging session i found that the pMainWnd has some special meaning but i could figure what exactly it has to do with modal windows (there is an undocumented "CWinApp::DoEnableModeless" for example).

EDIT: I'm posting a WM_CLOSE to the dialog and then use EndDialog(0) from the OnClose() handler to exit the modal state. I also tried to use EndDialog(0) directly. There is no difference between this two methods.

Answer 1

When MFC creates a modal dialog, it makes it modal by disabling the windows above it. The code that reenables those windows occurs when the dialog ends normally with a call to EndDialog. If anything prevents that code from running, the other windows will be locked out.

Modeless dialogs are a different beast, and there's a note specifically in the EndDialog documentation warning you to use DestroyWindow instead.

Answer 2

Maybe this is justifiable but I have a question:

why are you using hidden window? Was it created as message only window (passing HWND_MESSAGE as a parent handle and Message as a class) or you just call it message only?

OK, a little more info about MFC and dialogs.

MFC does not use Windows modal dialog. It always creates modeless dialog; either Create or DoModal call in turn ::CreateDlgIndirect windows API.

Modeless dialof rely on the main window message dispatch, while modal calls RunModalLoop that is similar to MFC window message pupmp (not a message loop). It runs in the main thread of execussion without freezing because it allows for idle processing (calls OnIdle).

How do you dismiss the modeless dialog? As Mark pointed you should use DestroyWindow.

As for m_pMainWnd, MFC framework uses it extensively to determine may things that control main window behavior. By changing it you may have created the behavior you experience.

Did you set the value to a newly created frame you treat as a main window?

What kind of MFC application is it? SDI or MDI? Would it be possible to create test app to duplicate this behavior and post it somewhere for download?

By the way, you do not have to be concern about DoEnableModeless, since it does not do anything but calls hook (COleFrameHook type) that is spasly used, unless you are trying to implement some functionality using OLE or ActiveX or you are trying to marry MFC and .NET Windows Forms. In conclusion if your (or third party code uses this hook, I would suggest checking the code in the COleFrameHook class.