[Mono][AOT] Deadlock in `mono_aot_get_class_from_name` when AOT assembly contains more than 65,000 classes

[andrew-kulikov]

Description

We faced an issue in our relatively large application having 1M+ LOC, most of them in one assembly. After some point in time, Android and iOS AOT build started hanging on app start. Removing some code or moving it to another assembly helped.

I investigated the problem and found a root cause in mono runtime:

A critical deadlock occurs in Mono AOT runtime when processing assemblies with a large number of classes (>65,000). The issue causes application hangs on both Android and iOS platforms using .NET 9 with AOT compilation.

The issue was introduced by PR #85952, which added an optimization to prevent failures when an AOT assembly contains more than 65,000 classes. When this threshold is exceeded, the AOT compiler produces a name table of size 0.

In the runtime code at aot-runtime.c:2711-2712, when the name table size is 0, the code path returns early without calling amodule_unlock. This causes the AOT module lock to be held indefinitely, leading to deadlocks when other threads attempt to access the same module.

Reproduction Steps

• Create a MAUI app
• AOT assembly contains more than 65,000 classes
• Multiple threads accessing the same AOT module (in my sample first thread in MainActivity, then main/gl thread in MainApplicaiton)
• One thread enters the early return path in mono_aot_get_class_from_name with size 0 name table

Reproduction sample could be found at https://github.com/andrew-kulikov/runtime-samples
Steps to reproduce:
1.

cd ./maui-aot-hang

2. Generate 65000 types in MauiAotHang.Features lib:

python ./generate_classes.py -o ./MauiAotHang.Features/sample.cs simple -n 65000

3. Publish app with mono AOT (for example for Android x64 emulator on windows)

dotnet publish ./MauiAotHang/MauiAotHang.csproj -f net9.0-android -r android-x64

4. Install and run app

Expected behavior

App works

Actual behavior

App gets stuck forever

Regression?

No response

Known Workarounds

Split large assembly to multiple assemblies of smaller size.

Configuration

• Runtime: Mono AOT
• OS: Android, iOS
• .NET Version: .NET 9
• Architecture: Android (x86_64, arm64-v8a), iOS (arm)
• Build Configuration: Release
• JIT Builds: Work correctly (issue is AOT-specific)

Other information

No response