System.OutOfMemoryException when allocating jump stub on macOS ARM64

[k15tfu]

Hi!

Sometimes I get a System.OutOfMemoryException exception in .NET 7 app on macOS ARM64 in arbitrary places like:

System.OutOfMemoryException
at InvokeStub_ContextConsumersWithAttributesManager..ctor(Object, Object, IntPtr*)
at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Digging into this, it turned out that the exception is thrown from HostCodeHeap::InitializeHeapList():

runtime/src/coreclr/vm/dynamicmethod.cpp

Lines 442 to 449 in d037e07

	pTracker = AllocMemory_NoThrow(0, JUMP_ALLOCATE_SIZE, sizeof(void*), 0);
	if (pTracker == NULL)
	{
	// This should only ever happen with fault injection
	_ASSERTE(g_pConfig->ShouldInjectFault(INJECTFAULT_DYNAMICCODEHEAP));
	delete pHp;
	ThrowOutOfMemory();
	}

because ExecutableAllocator::Commit() failed to commit executable memory:

runtime/src/coreclr/vm/dynamicmethod.cpp

Lines 758 to 762 in d037e07

	if (NULL == ExecutableAllocator::Instance()->Commit(m_pLastAvailableCommittedAddr, sizeToCommit, true /* isExecutable */))
	{
	LOG((LF_BCL, LL_ERROR, "CodeHeap [0x%p] - VirtualAlloc failed\n", this));
	return NULL;
	}

On macOS ARM64 ExecutableAllocator::IsDoubleMappingEnabled() is disabled, thus it internally calls VirtualAlloc(..., MEM_COMMIT, PAGE_EXECUTE_READWRITE) for the pre-reserved memory HostCodeHeap::m_pLastAvailableCommittedAddr, but then mprotect(..., PROT_EXEC | PROT_READ | PROT_WRITE) fails with errno EACCES (aka Permission denied):

runtime/src/coreclr/pal/src/map/virtual.cpp

Lines 1202 to 1216 in d037e07

	if (protectionState != vProtect)
	{
	// Change permissions.
	if (mprotect((void *) StartBoundary, MemSize, nProtect) != -1)
	{
	memset(pInformation->pProtectionState + runStart,
	vProtect, runLength);
	}
	else
	{
	ERROR("mprotect() failed! Error(%d)=%s\n",
	errno, strerror(errno));
	goto error;
	}
	}

because initially this memory region was reserved w/o MEM_RESERVE_EXECUTABLE flag (which is used for MAP_JIT):

runtime/src/coreclr/utilcode/util.cpp

Lines 448 to 460 in d037e07

	if ((mbInfo.State == MEM_FREE) &&
	(mbInfo.RegionSize >= (SIZE_T) dwSize || mbInfo.RegionSize == 0))
	{
	// Try reserving the memory using VirtualAlloc now
	pResult = (BYTE*)ClrVirtualAlloc(tryAddr, dwSize, MEM_RESERVE, flProtect);
	// Normally this will be successful
	//
	if (pResult != nullptr)
	{
	// return pResult
	break;
	}

despite the fact that ExecutableAllocator::ReserveWithinRange() actually requests it:

runtime/src/coreclr/utilcode/executableallocator.cpp

Lines 595 to 605 in d037e07

	else
	{
	DWORD allocationType = MEM_RESERVE;
	#ifdef HOST_UNIX
	// Tell PAL to use the executable memory allocator to satisfy this request for virtual memory.
	// This will allow us to place JIT'ed code close to the coreclr library
	// and thus improve performance by avoiding jump stubs in managed code.
	allocationType |= MEM_RESERVE_EXECUTABLE;
	#endif
	return ClrVirtualAllocWithinRange((const BYTE*)loAddress, (const BYTE*)hiAddress, size, allocationType, PAGE_NOACCESS);
	}

Linked issues: #50391, #54954.

[k15tfu]

I have checked that the issue is gone if we use MEM_RESERVE_EXECUTABLE flag, here is a draft PR #83819.

cc @janvorli