Memory leak in .Net 8 using Entity Framework and Sqllite (.Net Android)

[SebastianKleu]

Android framework version

net8.0-android, net9.0-android

Affected platform version

.NET 8 & .NET 9

Description

Hi,

We have been running our .Net Maui Android app on .Net 7 for some time now and upgraded to .Net 8.

After the upgrade, we realized that the app crashes after a few hours of use.

I have narrowed down the culprit to EF call on Sqllite. I am able to induce this problem by just reading from a table. See below:

private async Task TesterLoop()
    {
        while (true)
        {
            var testDataMessages = await GetTestDataMessages(new CancellationToken());
        }
    }

private async Task<List<TestData>> GetTestDataMessages(CancellationToken cancellationToken)
    {
	await using var dbContext = await _dbContextFactory.CreateDbContextAsync(cancellationToken);

        List<TestData> returnResult = await dbContext.TestData.AsNoTracking()
            .ToListAsync(cancellationToken);

        await dbContext.DisposeAsync().ConfigureAwait(false);

        return returnResult;
 }

Please see attached zip of test application where you can clearly see the memory rise in .Net 8 and not using .Net 7.

See below memory usage of .Net 7 app after 10 min.

See below memory usage of .Net 8 app after 10 min.

See below test application.

TestMaui.zip

Steps to Reproduce

1. Run Test.Maui.7 on Android device
2. Notice memory stays constant
3. Run Test.Maui.8 on Android device
4. Notice native memory climb fast

Did you find any workaround?

No

Relevant log output

No response

[SebastianKleu]

Just an update.
I was able to narrow down even more by completely removing the Maui UI from the test applications.
With the Maui UI removed, the memory still climbs rapidly for .Net 8 Android projects.
See updated test app:
TestMaui.zip

See results below:
Running Test.Android.7 for 3 min

Running Test.Android.8 for 3 min

[roji]

Note dotnet/efcore#34836 which tracks this on the EF side. As the problematic memory behavior appears when upgrading from .NET 7 to 8 - without any EF changes - we're assuming this is a non-EF problem for now... But please let me know if help is needed from the EF side!

[julealgon]

@roji any chance of this being related to that other problem with async TransactionScopes?

• GC doesn't run when using TransactionScope + async #50683
• Memory is not released when using TransactionScope with async option #108447

[roji]

@julealgon #50683 indeed seems like a duplicate of this. However, as with many "GC isn't working" issues that users open, I can't see any evidence of a leak - the complaint is that memory usage is growing, not that there's an actual OutOfMemoryException. So it may very well be that TransactionScopeAsyncFlowOption.Enabled causes increased memory usage, but that by no means implies a leak - the GC may simply not be reclaiming the memory yet (but would as memory usage approaches the limit). This is a point we keep need to explain to users.

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @roji, @ajcvickers
See info in area-owners.md if you want to be subscribed.

[roji]

@akoeplinger @jeffschwMSFT who's the right person to look at this? This is currently labeled as System.Data; the repro uses TransactionScope (which is System.Transactions rather than System.Data), but in any case at this point it seems like there's a memory-related issue on android only, which is why this was in the dotnet/android repo.

[vitek-karas]

@BrzVlad for ideas

[BrzVlad]

@SebastianKleu @roji Could you quickly port this to a console application? It is fine if it not going to leak, since it would be using CoreCLR, but I could test it with mono and it should be easier to investigate.

[SebastianKleu]

TestMaui.zip
@BrzVlad - Please see updated test application.
It now includes the following projects:
Test.Maui.7 -> DotNet 7 Maui app (no leak)
Test.Maui.8 -> DotNet 8 Maui app (leak detected)
Test.Android.7 -> DotNet 7 Android app (no leak)
Test.Android.8 -> DotNet 8 Android app (leak detected)
Test.Console.8 -> DotNet 8 Console app (no leak)

Please let me know if there's anything else I can provide to assist with the investigation.

[BrzVlad]

@SebastianKleu Seems like you uploaded the wrong archive since it doesn't contain the Console project.

[SebastianKleu]

@BrzVlad - Sorry about that. Seems I have reached the size limit of .zip folders for Github.

See link with updated test applications here: https://github.com/SebastianKleu/TestMaui

[markples]

I've been looking at TransactionScope (and ConditionalWeakTable and finalizers) and have at least a partial explanation. I'll try to write it up later today.

[markples]

#50683 contains my notes for ConditionalWeakTable (CWT). I don't understand this code enough to say whether CWT is a complete, partial, or irrelevant explanation for this case.

There are two aspects of CWT that may be relevant.

• The internal use of finalization can lead to very delayed collection of internal objects used by CWT - basically needing gen2 collections to make progress, so anything that the apps does to trigger gen2s (via allocation or manually) helps mitigate it.
• The finalization queue can get overloaded (entries added faster than they can be processed). For CWT this is probably only going to show up in microbenchmarks or if many threads are generating finalizable objects. Otherwise, it doesn't take much time where the app is doing non-finalization-creating work for the queue to drain.

[BrzVlad]

@SebastianKleu Note that I haven't been able to reproduce the leak on desktop so this issue would appear to be specific to android.

[SebastianKleu]

@BrzVlad - Yes, during my testing as well, it seems to only happen on Android devices.

[SebastianKleu]

Hi guys,

Is there anything we can do in the meantime to get around this problem? We really want to upgrade to the LTS version in .Net 8 but this issue is preventing us from doing so.

I would appreciate if there is some workarounds we can implement?

[SebastianKleu]

Hi guys,

Any update on this?

[BrzVlad]

Unfortunately I cannot reproduce this issue. I cloned https://github.com/SebastianKleu/TestMaui. Test.Android.8 doesn't leak for me (I included in the manifset <profileable android:shell="true" />, then built the app, opened the apk in android studio and used View Live Telemetry profiler option. App stayed stable at around 150 MB used memory). Test.Maui.8 crashes shortly after startup with MonoDroid: Java.Lang.RuntimeException: Context.startForegroundService() did not then call Service.startForeground(): ServiceRecord{e652e1f u0 com.test.maui.x8/com.maui.eight c:com.test.maui.x8} which seems like a problem with the sample.

I have 8.0.303 installed and was running on a Pixel 7a with android 14.

[GendacAI]

I am working with @SebastianKleu on the project. We are able to reproduce the issue on an Android 10 device. Android 12 and 13 seems to not have the issue. Running the app on an emulator also does not produce the memory issues.

The information for the device we are getting the issue looks as follow:

System Information:

CPU Information:

OS Information:

[BrzVlad]

Could you run the application with the MONO_LOG_LEVEL=debug and MONO_LOG_MASK=gc env vars set ? There should be some additional logging about managed memory usage with each GC.