Patch skipping crashing in functions marked with [UnmanagedCallersOnly]

[markoostveen]

Description

This is a repost of my stackoverflow post, that I have found a temporary solution for here

C# function pointers causes access violation upon calling from C++

I have tried debugging, but when the issue occurs. In this case C++ calls the callback, but before it enters the function's scope the exception is thrown, but immediately returns with an error code.

For the sake of debugging, I've tried switching the calling convention from cdecl to stdcall to see if it has to do with stack corruption.

My assumptions are that:

• static functions marked with [UnmanagedCallersOnly] have a static address, especially when an entrypoint is defined.
• Thread.BeginThreadAffinity() followed by Thread.EndThreadAffinity() ensures that an unmanaged thread can successfully start executing managed code.

Good to know:

• The function is being invoked by a thread not created in C#, there are many threads calling this function

Reproduction Steps

In the following bit of code, I'm using C# function pointers to omit using a delegate type, for faster performance. The code works fine when I use a delegate type marshalled as a function pointer, no exception no issues. However, when changing it from a delegate to a function pointer it sometimes throws an access violation.

Code is modified to remove namespaces and prefixes not relevant to the question

    public static class Engine
    {
        // Causing issues with sometimes throwing access violation
        [DllImport("engine", CallingConvention = CallingConvention.Cdecl)]
        public static unsafe extern UInt32 EventScheduler_ScheduleLocalEvent(
            int priority, UInt64 delay, delegate* unmanaged[Cdecl]<nint, void> callback, nint obj);

        // Works fine every time, but is much slower seems due to delegate instatiations
        [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
        public delegate void EventScheduler_ScheduleLocalEvent_callbackDelegate(nint obj);
        [DllImport("engine", CallingConvention = CallingConvention.Cdecl)]
        public static unsafe extern UInt32 EventScheduler_ScheduleLocalEvent(
            int priority,
            UInt64 delay,
            [MarshalAs(UnmanagedType.FunctionPtr)] EventScheduler_ScheduleLocalEvent_callbackDelegate callback,
            nint obj);

    }

    public static class EventScheduler
    {
        public delegate void EventCallback();

        // attribute is only present in the function pointer senario
        [UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl ) }, EntryPoint = "ScheduleLocalEvent_EventCallbackFunction")]
        private static void ScheduleLocalEvent_EventCallbackFunction(nint handlePtr)
        {
            Thread.BeginThreadAffinity();
            GCHandle handle = GCHandle.FromIntPtr(handlePtr);
            var cb          = (EventScheduler.EventCallback)handle.Target!;
            cb();
            handle.Free();
            Thread.EndThreadAffinity();
        }
        public static UInt32 ScheduleLocalEvent(int priority, SimulationTime delayTime, EventCallback eventCallback)
        {
            GCHandle handle  = GCHandle.Alloc(eventCallback, GCHandleType.Normal);
            IntPtr handlePtr = GCHandle.ToIntPtr(handle);
            unsafe
            {
                return Engine.EventScheduler_ScheduleLocalEvent(
                    priority, delayTime, &ScheduleLocalEvent_EventCallbackFunction, handlePtr);
            }
        }
   }

On the C++ side

    #define API extern "C" __declspec(dllexport)
    
    API uint32_t EventScheduler_ScheduleLocalEvent(int priority, uint64_t delay, void(__cdecl* callback)(void* obj), void* obj)
    {
        ERSAssert(Ers::Core::InsideSubModel());
        Core::SyncManagerBase& syncManager = Ers::Core::GetSyncManagerBase();
        return syncManager.ScheduleEvent(priority, delay, [callback, obj]() { callback(obj); });
    }

Expected behavior

The thread should enter and execute the static function passed using a function pointer

Actual behavior

The thread doesn't enter the user-defined portion of the function. an Access Violation is thrown in a generated call visible in the IL during the native-to-managed-transition

Regression?

No response

Known Workarounds

Change the garbage collection mode by adding ```xml
true
true

### Configuration

Tested on PC's around the office with the same behavior. All were x64, issue occurred on both Windows and Linux. The issue was reproduced on .net8.0 and .net9.0.100-preview

### Other information

The issue doesn't appear always, i think it might be a race condition inside the garbage collector.

[hez2010]

Can you try changing GCHandle.Alloc(eventCallback, GCHandleType.Normal) to GCHandle.Alloc(eventCallback, GCHandleType.Pinned)?

[jkotas]

Can you try changing GCHandle.Alloc(eventCallback, GCHandleType.Normal) to GCHandle.Alloc(eventCallback, GCHandleType.Pinned)?

This may hide issue similar to how changing the GC mode hides the issue, but it won't fix the underlying problem.

Thread.BeginThreadAffinity() followed by Thread.EndThreadAffinity() ensures that an unmanaged thread can successfully start executing managed code.

Thread.Begin/EndThreadAffinity do nothing in .NET Core. You can delete these calls. (These calls were relevant in .NET Framework for low-level hosting, e.g. under MS SQL Server.)

I have tried debugging, but when the issue occurs. In this case C++ calls the callback, but before it enters the function's scope the exception is thrown, but immediately returns with an error code.

Could you please share more details about the location of the crash? Where is the origin exception thrown exactly?

[markoostveen]

Can you try changing GCHandle.Alloc(eventCallback, GCHandleType.Normal) to GCHandle.Alloc(eventCallback, GCHandleType.Pinned)?

I've tried this before posting on StackOverflow, this didn't fix the issue. Also seemed to have no effect on the frequency it was occuring

Thread.Begin/EndThreadAffinity do nothing in .NET Core. You can delete these calls. (These calls were relevant in .NET Framework for low-level hosting, e.g. under MS SQL Server.)

Got it, that is nice to know!

Could you please share more details about the location of the crash? Where is the origin exception thrown exactly?

Yes, this is a screenshot where the exception get's thrown, though it only get's visibly thrown in .net9.0 preview in .net8.0 is calls abort and exists with an exit code indicating the Access Violation.
A portion is hidden as I wasn't allowed to show call paths from our project

This issue though always seems to happen in this call in the disassembly which I assume has something to do with setting up garbage collection

[jkotas]

This issue though always seems to happen in this call in the disassembly which I assume has something to do with setting up garbage collection

Yes, this call is unmanaged->managed transition. It would be useful to know where the crash happens inside this call. I do not think that the Visual Studio will show it to you. You would need to run under low-level debugger like windbg or collect crash dump using (https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps).

[markoostveen]

This issue though always seems to happen in this call in the disassembly which I assume has something to do with setting up garbage collection

Yes, this call is unmanaged->managed transition. It would be useful to know where the crash happens inside this call. I do not think that the Visual Studio will show it to you. You would need to run under low-level debugger like windbg or collect crash dump using (https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps).

Now this is where it gets interesting, while running it inside windbg the issue doesn't occur at all after running it many times, i'll try to get you a crash dump, any special settings you would like to be turned on for the minidump flags?

[jkotas]

any special settings you would like to be turned on for the minidump flags?

Default settings should work fine, to start with at least.

[markoostveen]

any special settings you would like to be turned on for the minidump flags?

Default settings should work fine, to start with at least.

Here you go
AWealthOfRows.exe.11692.dmp

[jkotas]

Thank you for sharing the minidump. Here are the details of the crash:

00007ff9`9f1abbea 833d07ed2e0000  cmp     dword ptr [00007ff9`9f49a8f8],0 <- ACCESS_VIOLATION here
00007ff9`9f1abbf1 7405            je      Ers!Ers.EventScheduler.ScheduleLocalEvent_EventCallbackFunction()+0x38 (00007ff9`9f1abbf8)
00007ff9`9f1abbf3 e8c80ac75f      call    coreclr!JIT_DbgIsJustMyCode (00007ff9`fee1c6c0)

The crash is in "Just My Code" check. It is an extra code that gets inserted by the JIT at the start of every method when running under managed debugger like Visual Studio. It allows the managed debugger to provide better stepping experience. It explains why you are not able to reproduce the crash outside Visual Studio.