Managed code in a hosted .NET Core runtime corrupts the caller stack

Question

I am playing with the .NET Core runtime hosting example.

When I build it "as is" (64 bit) on Windows, it works.

When I build it as a 32-bit application (on Windows) and change the .NET runtime to x86 (-r win-x86), it crashes.

This is what's happening. After the managedDelegate returns, the stack of the caller (main()) is corrupted and the app crashes.

doWork_ptr managedDelegate;

createManagedDelegate(
        hostHandle,
        domainId,
        "ManagedLibrary, Version=1.0.0.0",
        "ManagedLibrary.ManagedWorker",
        "DoWork",
        (void**)&managedDelegate);

char* ret = managedDelegate("Test job", 5, sizeof(data) / sizeof(double), data, ReportProgressCallback);

When I change the managed method (DoWork) to a void-returning one without any parameters, it works.

It seems that I am missing something about the calling conventions, but cannot spot what exactly. The default one is stdcall on Windows, but there are some differences between x86 and x64 too. x64 uses a special x64 fastcall convention, and I suspect it somehow messes up the whole thing when hosting the .NET CLR in a 32-bit app.

What do I need to change to get this running? Do I need to build the native (host) app using a specific calling convention? Do I need to decorate the managed methods with special attributes? Or maybe somehow configure the hosted .NET Core runtime?

Answer 1

As @HansPassant mentioned in the comments:

The declaration of the function pointers is critical, for x86 you get to deal with incompatible calling conventions. There is no distinction between cdecl and stdcall in x64. More macro soup needed, the Windows declaration would be typedef int (__stdcall *report_callback_ptr)(int progress);, etc.

That is the trick.

The callback and the managed method function pointer need to be additionally decorated with the __stdcall attribute:

typedef int (__stdcall *report_callback_ptr)(int progress);
typedef char* (__stdcall *doWork_ptr)(const char* jobName, int iterations, int dataSize, double* data, report_callback_ptr callbackFunction);

The callback implementation needs to be decorated too:

int __stdcall ReportProgressCallback(int progress) { /* implementation here */ }

It turns out that the managed marshaler treats the managed methods as __stdcall on x86, which is no surprise.

Applying those changes makes the sample work when built as an x86 app hosting the x86 .NET Core runtime.