Function pointer incorrect in Visual Studio 2005, code starts at 1 byte offset

Question

The code in question hooks into explorer.exe but was crashing on entry to the callback function:

Unhandled exception at 0x60055b50 (redacted.dll) in explorer.exe: 0xC0000005: Access violation writing location 0x548b0cca.

Callstack:

>     redacted.dll!myCallWndProcRetCallback(int nCode=0x00000000, unsigned int wParam=0x00000000, long lParam=0x015afa58)  Line 799  C++
      user32.dll!_DispatchHookW@16()  + 0x31 bytes    
      user32.dll!_fnHkINLPCWPRETSTRUCTW@20()  + 0x5e bytes  
      user32.dll!___fnDWORD@4()  + 0x24 bytes   
      ntdll.dll!_KiUserCallbackDispatcher@12()  + 0x13 bytes      
      user32.dll!_NtUserMessageCall@28()  + 0xc bytes 
      user32.dll!_SendMessageW@16()  + 0x49 bytes     
      explorer.exe!CTaskBand::_FindIndexByHwnd()  + 0x21 bytes    
      explorer.exe!CTaskBand::_HandleShellHook()  + 0x48 bytes    
      explorer.exe!CTaskBand::v_WndProc()  + 0x660 bytes    
      explorer.exe!CImpWndProc::s_WndProc()  + 0x3f bytes

Visual Studio 2005 gave the following disassembly:

--- c:\projects\redacted.cpp -------------------------
//------------------------------------------------------------------------------
LRESULT CALLBACK myCallWndProcRetCallback(int nCode, WPARAM wParam, LPARAM lParam) {
60055B50  inc         dword ptr [ebx+548B0CC4h] 
60055B56  and         al,18h 
60055B58  mov         eax,dword ptr [g_callWndProcRetHook (600B9EE8h)] 
60055B5D  push        esi

and the memory around 0x548B0CC4 is all ?????? so it is not mapped memory, hence the crash.

The machine code at the start of myCallWndProcRetCallback is this:

0x60055B50:  ff 83 c4 0c 8b 54 24 18 a1 e8 9e 0b 60 56 52 57 50 ff 15 8c a6 09 60 5f 5e 83 c4 08 c2 0c 00 cc 8b 4c 24 04 8b 01 8b 50

But Visual Studio also sometimes gives the following disassembly for this function:

--- c:\projects\redacted.cpp -------------------------
60055B51  add         esp,0Ch 
    if ( nCode == HC_ACTION && lParam != NULL) {
60055B54  mov         edx,dword ptr [esp+18h] 
60055B58  mov         eax,dword ptr [g_callWndProcRetHook (600B9EE8h)] 
60055B5D  push        esi

This looks like the correct disassembly but it is from 1 byte later than the disassembly above! You can see the instructions are the same from 0x60055B58 onward.

So, it looks like the linker says the function is at 0x60055B50 but the code actually starts at 0x60055B51. I have confirmed the former is the callback set into the Windows hook. So when Windows calls back into the function it executes bad code.

The question I have is how the linker could get this wrong? I did a rebuild and the problem went away, it seems random. At the time the /FORCE:MULTIPLE linker option was in effect but without it no link error is reported for this callback.

A late addition: Could this be related to the relocation or rebasing of a DLL? If the relocation was off by 1 byte, this could perhaps cause the problem?

MSN MSN · Accepted Answer · 2009-01-22T23:59:27

Relocations will almost never be off by 1 byte; the .dll image has to be aligned to the granularity of allocations returned by VirtualAlloc which should be 64k on most machines.

How long has this code worked? If it's random then the /FORCE:MULTIPLE might be suspect. Or you could be using Incredibuild...

Function pointer incorrect in Visual Studio 2005, code starts at 1 byte offset

1 Answers