Calling C++ dll from C#. “Cannot marshal 'return value': Invalid managed/unmanaged type combination.”

Question

My Header file.

extern "C"  class MyFuncs
{
 public:
    __declspec(dllexport) unsigned char PassImage(unsigned char buffer, int size);
};

CPP file.

unsigned char MyFuncs::PassImage(unsigned char buffer, int size)
{
    return buffer;
}

All works well except when i am returning buffer back to my main application.

    [DllImport("ExampleDLL.dll", EntryPoint = "?PassImage@MyFuncs@Funcs@@QAEXEH@Z")]
    public static extern byte[] PassImage(byte[] a, int count);

The error occurs when i am return unsigned char to byte[].

If i change byte[] to byte i get a value back an no error.

This is the exact error:

Cannot marshal 'return value': Invalid managed/unmanaged type combination.

How can I accept back unsigned char to byte[]?

Something must be off in the code example you posted. an unsigned char is not an array. Maybe that should be unsigned char * ? — Moby Disk
Beyond the erroneous use of unsigned char, you can't return a byte[] like that. — David Heffernan
And I repeat, you cannot return a byte[] like that. What we need to understand is what you are trying to do. You also need to get on top of what unsigned char is. It is a single scalar. It's not an array. Oh, and you also have an instance method which is not good too. So, it's just a total mess at the moment. You need to step back, go more slowly, think more deeply. Thrashing around won't get you anyway. Seek understanding. — David Heffernan

Fritschifisch Fritschifisch · Accepted Answer · 2015-02-09T21:10:55

The unsigned char should be an unsigned char * :)

EDIT: You also need to pass the length of the array and handle it as a byte pointer in C# because .NET doesn't know the length of it. Described in this article: https://stackguides.com/questions/8268625/get-pointer-on-byte-array-from-unman‌aged-c-dll-in-c-sharp