Why does this simple DLL crash before WM_ACTIVATEAPP arrives?

Question

The minimal DLL below, which uses only the Win32 API, tries to do nothing more than create an MDI frame/client window and one child window, and destroy the frame window when the DLL unloads. The DLL crashes on Windows XP with an exception upon executing an INT x2B instruction in USER32.

For testing, the DLL is simply invoked by a one-line application calling LoadLibrary('badcode.dll').

The crash happens inside the final "DestroyWindow(framewindowhandle)" just before the DLL finishes, after FrameWindowProc receives WM_ACTIVATE but before it receives WM_ACTIVEAPP.

The DLL code has been trimmed down from a much larger original as much as possible to isolate the bug. Although not destroying the frame window also makes the current crash go away, about 12 years ago it was determined that tools like Visual Basic running on NT would crash unless "DestroyWindow(framewindowhandle)" was called before the DLL was unloaded. Just recently, however, a new small program written to test some of the DLL entrypoints was suddenly found to be crashing on XP as described above.

Although written in Delphi 6, the code only relies on the vanilla Win32 API.

library badcode; // works if rewritten as a program instead of DLL

{$R *.RES} // removing this avoids crash

uses windows, messages; // only win32 calls are made

// 3 MDI window handles
var framewindowhandle, clientwindowhandle, childwindowhandle: hwnd;

function framewindowproc(windowhandle: hwnd; message: word; wparam, lparam: longint): longint; stdcall;
var ccs: tclientcreatestruct;
begin // frame window has received a message
if message = WM_CREATE then
  begin // create the client window
  ccs.hwindowmenu := 0; ccs.idfirstchild := 0;
  clientwindowhandle := createwindow('MDICLIENT', '', ws_child + ws_clipchildren + ws_visible, 10, 10, 50, 50, windowhandle, 0, hinstance, @ccs);
  result := 0; // we handled the message
  end
else // do default handling
  result := defframeproc(windowhandle, clientwindowhandle, message, wparam, lparam);
end;

function childwindowproc(windowhandle: hwnd; message: word; wparam, lparam: longint): longint; stdcall;
begin // child window has received a message, do default handling
result := defmdichildproc(windowhandle, message, wparam, lparam);
end;

procedure DLLHandler(reason: integer);
begin
if reason = DLL_PROCESS_DETACH then // unloading dll
  DestroyWindow(framewindowhandle); // causes the crash, never returns
end;

var wc: twndclass; mcs: tmdicreatestruct;

begin // DLL loading time
DLLProc := @DLLHandler; // so we can detect unload
wc.hinstance := hinstance;
wc.lpfnwndproc := @framewindowproc;
wc.style := 0; wc.cbclsextra := 0; wc.cbwndextra := 0;
wc.hicon := loadicon(0, IDI_ASTERISK);
wc.hcursor := loadcursor(0, IDC_ARROW);
wc.hbrbackground := 0;
wc.lpszmenuname := 'MENUBAR'; // changing to '' avoids the crash
wc.lpszclassname := 'BAD';
registerclass(wc); // register the frame window

wc.lpfnwndproc := @childwindowproc;
wc.lpszmenuname := '';
wc.lpszclassname := 'DATA';
registerclass(wc); // register the child window

framewindowhandle := createwindow('BAD', 'frame', WS_OVERLAPPEDWINDOW + WS_CLIPCHILDREN, 100, 100, 400, 600, 0, 0, hinstance, nil);

mcs.szclass := 'DATA'; mcs.sztitle := 'child'; mcs.howner := hinstance;
mcs.x := 50; mcs.y := 50; mcs.cx := 50; mcs.cy := 50; mcs.style := WS_MINIMIZE; // changing the style avoids the crash
childwindowhandle := sendmessage(clientwindowhandle, WM_MDICREATE, 0, longint(@mcs));
sendmessage(clientwindowhandle, WM_MDIRESTORE, childwindowhandle, 0); // skipping this avoids the crash
end.

The different behavior of different OS can be due to implementation details of corresponding OS. What you're doing have always been considered unsafe. Consider the quote in the following comment from DllMain entry point — Sertac Akyuz
"Calling functions that require DLLs other than Kernel32.dll may result in problems that are difficult to diagnose. For example, calling User, Shell, and COM functions can cause access violation errors, because some functions load other system components. Conversely, calling functions such as these during termination can cause access violation errors because the corresponding component may already have been unloaded or uninitialized." — Sertac Akyuz

Witness Protection ID 44583292 Witness Protection ID 44583292 · Accepted Answer · 2012-12-16T02:34:18

Using the excellent dependencywalker tool, I discovered some old scanner software on my machine had configured USER32 to hook in an OCR-related DLL upon the execution of any program, and that DLL was making some questionable-looking calls, including being loaded twice for some reason. Uninstalling the scanner software made the crash go away and all O/S DLL loading/unloading look much more reasonable. Nevertheless, I'll be modifying my DLL to do nothing during attach/detach, and include new entrypoints for starting/stopping.

Why does this simple DLL crash before WM_ACTIVATEAPP arrives?

1 Answers