Calling managed code from unmanaged win32 assembly dll - crash

Question

I'm developing a serial port dll in win32 assembly (MASM32). It has its own thread checking multiple events and at a specified buffer threshold it'd notify the managed main application by calling a callback function. It is just a call with no arguments/return value.

At startup the main application stores the callback function's address by calling a function in the dll:

pCallBackFunction dd 0

SetCallBackPointer proc pcb:DWORD

mov eax, pcb
mov pCallBackFunction, eax
call DWORD ptr pCallBackFunction ; verify it immediately
ret

SetCallBackPointer endp

The upper function immediately calls back the managed application callback routine for verification purposes. It is working fine. However, when I place the call instruction to other functions in the dll it crashes the application. It doesn't matter if the call is in a simple function or in the threadproc of the dll. For example:

OpenPort proc pn:byte,br:dword, inputbuffersize: dword, outputbuffersize:dword, thresholdsize: dword
LOCAL dcb: DCB
LOCAL SerialTimeOuts: COMMTIMEOUTS
call DWORD ptr pCallBackFunction
xor eax, eax
mov al, pn
mov [com_port+3],al

etc. etc.

will crash at call DWORD ptr pCallBackFunction always. Since I call SetCallBackPointer first to store a valid address in pCallBackFunction, it should have a valid address.

My managed app is written in C# and the relevant part is:

public partial class Form1 : Form
{
    public delegate void CallBackDelegate();
    public static CallBackDelegate mydelegate;

    [DllImport("serialport.dll")]
    private static extern void SetCallBackPointer(CallBackDelegate Delegate);

    [DllImport("serialport.dll")]
    public static extern int OpenPort(byte com, uint br, uint inbufsize, uint outbufsize, uint threshsize);



    public Form1() 
    {

        InitializeComponent();



        mydelegate =new CallBackDelegate(CallbackFunction);

        SetCallBackPointer(mydelegate);

        unsafe
        {
            int sysstat;
            int hResult;

            hResult = OpenPort(Convert.ToByte('5'), 9600, 306, 4, 4);
        }
        }

        public static void CallbackFunction()
        {
             MessageBox.Show( "CallBack Function Called by Windows DLL");
        }

The VS debugger reported that the dll had tried to read/write from/to a protected memory address. But when calling SetCallBackPointer there is no such problem. What am I doing wrong here?

Any tips would be great!

If you're sure everything else is correct (pCallBackFunction won't be overwritten) did you check the calling convention (OpenPort has many parameters and a return value)? — Adriano Repetti
By the way...+1 because you're writing in assembly even if I can't figure the reason! — Adriano Repetti
OpenPort is working fine when I cooment out that call dword...instruction. I'll add extra functions to check against overwriting pCallBackFunction. It's just practicing, no real reasons behind writing in assembly. Thanks. — JustGreg
My apologies, found an uncommented line overwriting pCallBackFunction. It's working fine now. — JustGreg

Viktor Latypov Viktor Latypov · Accepted Answer · 2012-06-29T15:13:14

SetCallBackDelegate must use the IntPtr, not the Delegate object.

You must use the Marshal.GetFunctionPointerForDelegate function to convert the System.Delegate to IntPtr32. Otherwise you're passing some pointer which points somewhere to the managed heap.

So change the

 mydelegate =new CallBackDelegate(CallbackFunction);
 SetCallBackPointer(mydelegate);

to

 mydelegate =new CallBackDelegate(CallbackFunction);

 SetCallBackPointer( Marshal.GetFunctionPointerForDelegate(delegate) );

and fix the C# declaration of SetCallBackPointer to

[DllImport("serialport.dll")]
private static extern void SetCallBackPointer(IntPtr Delegate);

Calling managed code from unmanaged win32 assembly dll - crash

1 Answers