MulticoreJIT accesses m_GenericInfoCount field from multiple threads without synchronization

[agocke]

See MultiCoreJitRecord::WriteOutput for an example.

This looks like it can cause memory corruption.

[agocke]

@mangod9 It looks like this is the cause of the crashes in the PGO runs. It may also be a severe problem for other users of multi-core JIT.

cc @davidwrighton

[jkotas]

#39996 cc @clamp03

[mangod9]

@agocke do you have a link to the pgo jobs which are failing?

[agocke]

https://dev.azure.com/dnceng/internal/_build/results?buildId=939897&view=logs&j=4859eead-9322-5aef-9825-19eb054d3a82

Although I wouldn't suggest trying to debug through that -- I can provide a dump, but simply doing a volatile load from that field at the beginning of WriteOutput and using a local from then on was enough to reduce the race to no longer produce memory corruption.

I'm not sure what the appropriate level of synchronization for that field is, though.

[clamp03]

Hi, @agocke I cannot access that dump. In the dumped backtrace, is there "WriteMulticoreJitProfiler"?
I found that both recording and writeoutput can run if writeouput is executed from WriteMulticoreJitProfile in non-unix.
If not, could you please share the backtrace?

[agocke]

Yup, stack trace in the crash looks like

 	ntdll.dll!RtlReportFatalFailure�()	Unknown
 	ntdll.dll!RtlReportCriticalFailure�()	Unknown
 	ntdll.dll!RtlpHeapHandleError�()	Unknown
 	ntdll.dll!RtlpHpHeapHandleError�()	Unknown
 	ntdll.dll!RtlpLogHeapFailure�()	Unknown
 	ntdll.dll!RtlpFreeHeapInternal()	Unknown
 	ntdll.dll!RtlFreeHeap()	Unknown
>	coreclr.dll!ClrFree(void * p) Line 302	C++
 	coreclr.dll!operator delete[](void * p) Line 410	C++
 	coreclr.dll!MulticoreJitRecorder::WriteOutput(IStream * pStream) Line 557	C++
 	coreclr.dll!MulticoreJitRecorder::WriteOutput() Line 155	C++
 	coreclr.dll!MulticoreJitRecorder::StopProfile(bool) Line 951	C++
 	coreclr.dll!MulticoreJitRecorder::WriteMulticoreJitProfiler(_TP_CALLBACK_INSTANCE * pInstance, void * pvContext, _TP_TIMER * pTimer) Line 810	C++
 	ntdll.dll!TppTimerpExecuteCallback()	Unknown
 	ntdll.dll!TppWorkerThread()	Unknown
 	kernel32.dll!00007fff9a567034()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

[agocke]

@jkotas This manifested as memory corruption in the test case. Do we want to consider backporting the fix?

[jkotas]

#39996 that introduced this crash is in .NET 6 only.