Memory leak in .Net 8 using Entity Framework and Sqllite (.Net Maui Android)

[SebastianKleu]

Description

Hi,

We have been running our .Net Maui Android app on .Net 7 for some time now and upgraded to .Net 8.

After the upgrade, we realized that the app crashes after a few hours of use.

I have narrowed down the culprit to EF call on Sqllite. I am able to induce this problem by just reading from a table. See below:

private async Task TesterLoop()
    {
        while (true)
        {
            var testDataMessages = await GetTestDataMessages(new CancellationToken());
        }
    }

private async Task<List<TestData>> GetTestDataMessages(CancellationToken cancellationToken)
    {
	await using var dbContext = await _dbContextFactory.CreateDbContextAsync(cancellationToken);

        List<TestData> returnResult = await dbContext.TestData.AsNoTracking()
            .ToListAsync(cancellationToken);

        await dbContext.DisposeAsync().ConfigureAwait(false);

        return returnResult;
 }

Please see attached zip of test application where you can clearly see the memory rise in .Net 8 and not using .Net 7.

See below memory usage of .Net 7 app after 10 min.

See below memory usage of .Net 8 app after 10 min.

See below test application.
TestMaui.zip

Steps to Reproduce

1. Run Test.Maui.7 on Android device
2. Notice memory stays constant
3. Run Test.Maui.8 on Android device
4. Notice native memory climb fast

Link to public reproduction project repository

No response

Version with bug

8.0.91 SR9.1

Is this a regression from previous behavior?

Yes, this used to work in .NET MAUI

Last version that worked well

7.0.101

Affected platforms

Android

Affected platform versions

Android 10

Did you find any workaround?

No

Relevant log output

No response

[bengavin]

@SebastianKleu - can this be reproduced if you upgrade EFCore to 8.x? Any particular reason you're choosing to stay with 6.x?

[roji]

@SebastianKleu also, the code snippet above doesn't seem like it has anything to do with MAUI. Are you saying the same trouble e.g. in a console program? Can you submit a minimal non-MAUI console program that reproduces this? That would help the investigation process considerably.

[SebastianKleu]

@bengavin - Even if I upgrade to version 8, the memory leak still occurs.
@roji - I updated the test app to include Android apps without using MAUI UI. The memory leak is still happening on .net 8.
See updated test app:
TestMaui.zip
See results below:
Running Test.Android.7 for 3 min

Running Test.Android.8 for 3 min

[roji]

@SebastianKleu I can take a look, but are you seeing an actual OutOfMemoryException at some point? If not, this is probably the normal behavior of a garbage-collected language - the GC kicks in according to various criteria, e.g. only when memory is tight.

I'm not sure why things would have changed between 7 and 8, but it's quite unlikely to be related to EF in any way. You can verify this by using EF 7 from .NET 8 - that would isolate the EF version from the .NET version.

[SebastianKleu]

@roji - It does not matter which EF Core nuget package version I am using (I have tried 6.x.x, 7.x.x. and 8.x.x). All versions seems to not work on Dotnet 8 Android devices due to memory consumption. When using Dotnet Android 7 devices (tested with 6.x.x. and 7.x.x EF Core nuget packages) it seems to work just fine.

Viewing the logcat logs, I cannot see anything with regards to OutOfMemoryException, I can just see before 10min the app crashes with logs below:
10-08 14:17:45.411 18829 18917 W libc : calloc(1, 192) failed: returning null pointer
10-08 14:17:45.426 18829 18917 E : Could not allocate 192 (1 * 192) bytes
--------- beginning of crash
10-08 14:17:45.427 18829 18917 F libc : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 18917 (.NET Long Runni), pid 18829 (m.test.android8)
10-08 14:17:46.013 1160 1332 D CompatibilityInfo: mCompatibilityFlags - 0
10-08 14:17:46.013 1160 1332 D CompatibilityInfo: applicationDensity - 320
10-08 14:17:46.013 1160 1332 D CompatibilityInfo: applicationScale - 1.0
10-08 14:17:46.066 24015 24015 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone
10-08 14:17:46.075 827 827 I /system/bin/tombstoned: received crash request for pid 18917
10-08 14:17:46.083 24015 24015 I crash_dump32: performing dump of process 18829 (target tid = 18917)
10-08 14:17:46.178 24015 24015 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-08 14:17:46.178 24015 24015 F DEBUG : Build fingerprint: 'qcom/qcm2150/qcm2150:10/1.0.101/eng.sefeko.20240903.111753:userdebug/test-keys'
10-08 14:17:46.178 24015 24015 F DEBUG : Revision: '0'
10-08 14:17:46.178 24015 24015 F DEBUG : ABI: 'arm'
10-08 14:17:46.208 24015 24015 F DEBUG : Timestamp: 2024-10-08 14:17:46+0200
10-08 14:17:46.208 24015 24015 F DEBUG : pid: 18829, tid: 18917, name: .NET Long Runni >>> com.test.android8 <<<
10-08 14:17:46.208 24015 24015 F DEBUG : uid: 10109
10-08 14:17:46.208 24015 24015 F DEBUG : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
10-08 14:17:46.208 24015 24015 F DEBUG : r0 00000000 r1 000049e5 r2 00000006 r3 7ccb5e18
10-08 14:17:46.208 24015 24015 F DEBUG : r4 7ccb5e2c r5 7ccb5e10 r6 0000498d r7 0000016b
10-08 14:17:46.209 24015 24015 F DEBUG : r8 7ccb5e28 r9 7ccb5e18 r10 7ccb5e48 r11 7ccb5e38
10-08 14:17:46.209 24015 24015 F DEBUG : ip 000049e5 sp 7ccb5de8 lr ab77d4db pc ab77d4ee
10-08 14:17:46.226 24015 24015 F DEBUG :
10-08 14:17:46.226 24015 24015 F DEBUG : backtrace:
10-08 14:17:46.226 24015 24015 F DEBUG : #00 pc 000604ee /apex/com.android.runtime/lib/bionic/libc.so (abort+166) (BuildId: 1f4693f0a28e39adb613e92bac7a19b4)
10-08 14:17:46.226 24015 24015 F DEBUG : dotnet/android#1 pc 0001add1 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonodroid.so (xamarin::android::Helpers::abort_application()+4) (BuildId: 16249c43dc55845bac38ca3d64ee29221e3cf954)
10-08 14:17:46.227 24015 24015 F DEBUG : dotnet/android#2 pc 00031437 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonodroid.so (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+66) (BuildId: 16249c43dc55845bac38ca3d64ee29221e3cf954)
10-08 14:17:46.227 24015 24015 F DEBUG : dotnet/android#3 pc 0015923d /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.227 24015 24015 F DEBUG : dotnet/android#4 pc 0014fb65 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.227 24015 24015 F DEBUG : dotnet/android#5 pc 0014fbad /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.227 24015 24015 F DEBUG : dotnet/android#6 pc 0014f923 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#7 pc 001109fb /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#8 pc 001113ed /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#9 pc 0012c233 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#10 pc 00122851 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#11 pc 00120ea1 /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.228 24015 24015 F DEBUG : dotnet/android#12 pc 001b15eb /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (mono_runtime_invoke_checked+90) (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.229 24015 24015 F DEBUG : dotnet/android#13 pc 001c557d /data/app/com.test.android8-trFTisTVWk-8tJZYD9NJEQ==/lib/arm/libmonosgen-2.0.so (BuildId: 9e6f759694f443184938217050f2ad66b1f384ed)
10-08 14:17:46.229 24015 24015 F DEBUG : dotnet/android#14 pc 000aa8bb /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+20) (BuildId: 1f4693f0a28e39adb613e92bac7a19b4)
10-08 14:17:46.229 24015 24015 F DEBUG : dotnet/android#15 pc 00061993 /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30) (BuildId: 1f4693f0a28e39adb613e92bac7a19b4)

[roji]

If the deciding factor here is the .NET version (6 and 7 are fine, 8 isn't), and the EF version doesn't matter, then this is something you're going to have to take up with the MAUI team - there may have been some sort of change there for .NET 8 which explains this.

Keep in mind that increasing memory usage in itself does not indicate a memory leak - the GC decides when to reclaim memory based on its own set of considerations (e.g. when there's little memory left), and it's perfectly fine for memory usage to just keep increasing without getting reclaimed for a long while.

I don't question that there's a problem with your app on MAUI - there's indeed a clear error there about failing to allocate. At the same time, if there's a memory leak in EF, it would cause an OutOfMemoryException when running the program (not on MAUI), and as far as I understand that's not the case. If you can trigger and OutOfMemoryException outside of MAUI with a minimal code sample, that's definitely something I'll want to see.

[SebastianKleu]

@roji - I did reach out to the Maui team.
See bug here: dotnet/maui#25087
They then directed me to EF Core because we established that the issue is still happening even without Maui UI.
I also loaded a bug on dotnet android team: dotnet/runtime#108762
It definitely seems like an issue with dotnet 8 on Android devices. I hope they will be able to help me.
I'll keep a lookout for the OutOfMemoryException but we have yet to receive it in our logs.

[roji]

@SebastianKleu thanks for the issue references and the details.

I'll keep a lookout for the OutOfMemoryException but we have yet to receive it in our logs.

Note that the logs you pasted above indeed point to an out-of-memory error (on Android). My suggestion here is to try to reproduce the same problem on regular .NET (non-mobile), as a way of trying to isolate the issue - for that, I'd expect a OutOfMemoryException if there's an actual leak somewhere. In other words, if you can produce a vanilla .NET repro - not mobile, just a regular .NET console program - which triggers an OutOfMemoryException, that would be the ideal way to make progress with this issue. It's still very unlikely this has anything to do with EF (since the problem started after switching from .NET 7 to 8), but that's what's needed to narrow down the problem and provide the runtime people with the repro they need.

Otherwise, if the issue really is only reproducible on Android, things are trickier - and are probably best investigated in dotnet/runtime#108762. I'll post on that issue, and let's see what comes out of that.

[SebastianKleu]

@roji - Thank you.
I can confirm that this is not happening in a normal console application. It seems to only occur on Android devices.
My hope now is that somebody can help us out in dotnet/runtime#108762

[roji]

Thanks for confirming @SebastianKleu.

I'll go ahead and close this issue for now, as it's quite unlikely this is an actual EF issue. However, based on dotnet/runtime#108762 we can reopen as needed.