Cannot marshal 'return value': Invalid managed/unmanaged type combination

Question

I have this function in an unmanaged library and I want to call it in C#:

unsigned char __stdcall InitDev(unsigned char comport, long BaudRate)

and this is in C#

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("*.dll", CallingConvention = CallingConvention.Cdecl,CharSet =CharSet.Auto)]
public static extern byte[] InitDev(byte[] comport, [Out]long BaudRate);

but when I call this function in C# I get the following error

"Cannot marshal 'return value': Invalid managed/unmanaged type combination."

string COM = "COM3";
byte[] ptr = new byte[1];
try
{
    ptr = InitDev(Encoding.ASCII.GetBytes(COM), 9600);
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

I am new to this if a little guidance is provided I will be able to resolve this issue

This function is next to impossible to call reliably in a C program, that gets never better when you do it from C#. You don't know how many elements are in the array and you don't know how to release it. You'll have to talk to the programmer of this code to find out what that pointer means. As long as you do, get him to fix it. The baudrate argument is an int. — Hans Passant

Mateusz Mateusz · Accepted Answer · 2017-07-24T15:10:31

Assuming your native call is something like unsigned char* InitDef(unsigned char*, long long &baudRate) then you probably want to use IntPtr in the place of your unsigned char*. But first since it's managed/unmanaged mixing you should allocate that byte[] buffer for input and output :

byte[] pInBuffer = Encoding.ASCII.GetBytes("COM3");
// allocate unmanaged memory
IntPtr inputBuffer = Marshal.AllocHGlobal(pInBuffer.Length * sizeof(byte));
Marshal.Copy(pInBuffer, 0, inputBuffer, pInBuffer.Length);

// now your inputBuffer contains a native pointer to the `unsigned char*`
// and since your function returns a new pointer to some `unsigned char*`
// just retrieve it to IntPtr
IntPtr result = InitDev(inputBuffer, 9600);

// free your allocated memory
Marshal.FreeHGlobal(inputBuffer);

At this stage your result contains the value returned from InitDev.

new definition of InitDev function

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("*.dll", CallingConvention = CallingConvention.Cdecl,CharSet =CharSet.Auto)]
public static extern IntPtr InitDev(IntPtr comport, [Out] long BaudRate);

All of the necessary calls, you can find on this msdn page

EDIT :
Since your native call looks like this :extern unsigned char __stdcall InitDev(unsigned char comport,long BaudRate);
I'm guessing that the first parameter is just the last digit ( index ) of your "COMn" because unsigned char's lowest possible value is 0 and COM's lowest index can be 0.

Try using this snippet :

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("*.dll", CallingConvention = CallingConvention.Cdecl,CharSet =CharSet.Auto)]
public static extern byte InitDev(byte comport, long BaudRate);

And then call it like so :

byte comIdx = 3;
byte result = InitDev(comIdx, 9600);
// byte should yield correct result now

Cannot marshal 'return value': Invalid managed/unmanaged type combination

1 Answers