Very Poor ImageSharp Performance in MAUI Android

[david-maw]

Description

A piece of code which executes in under 100 ms in Xamarin forms on Android or Windows and under 50 ms in Windows MAUI takes over 30 seconds in Android MAUI.

Steps to Reproduce

1. Clone repository https://github.com/david-maw/ResizeImageMaui.git from github
2. Build the project
3. Run it on Windows
4. Click the button, on Windows the image will be converted to greyscale in under a second (50 ms in my case).
5. Try it again on Android, this time the conversion will take much longer (in my case it was over 30 seconds but less than a minute.

FYI there's a similar Xamarin Forms project at https://github.com/david-maw/ResizeImageXamarin.git, in my testing this took about 90 ms to load on Window (so MAUI was almost twice as fast), and just under a second on the Android emulator (so MAUI was 30+ times slower).

This uses SixLabors.ImageSharp to do the image processing so it is likely the implementation of something in that library that's wildly slow on .NET 6 on Android.

Version with bug

6.0 Release Candidate 3

Last version that worked well

Unknown/Other

Affected platforms

Android, I was not able test on other platforms

Affected platform versions

Android 11

Did you find any workaround?

No

Relevant log output

No response

[jfversluis]

@jonathanpeppers this seems like something for you :)

[janseris]

Did you try Release configuration for the benchmarking?

[jonathanpeppers]

@david-maw yes you should try a Release build, and maybe try "full" AOT:

<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
  <RunAOTCompilation>true</RunAOTCompilation>
  <AndroidEnableProfiledAot>false</AndroidEnableProfiledAot>
</PropertyGroup>

Release builds use profiled AOT by default, and SixLabors.ImageSharp would not be in our built-in profile.

[NonameMissingNo]

@jonathanpeppers
Mi 11i (Android):
Debug: 35.841 Seconds
Release: 7.212 Seconds (2nd run: 7.137 Seconds)
Release with full AOT: 7.151 Seconds

For reference Windows (Ryzen 5 5500U):
Debug: 210 ms
Release: 165 ms

[NonameMissingNo]

Release is a lot better, but I don't think it should be 43 times slower than the windows Release build

[jonathanpeppers]

@NonameMissingNo did you already get a .speedscope file, to see where the problem is?

https://github.com/xamarin/xamarin-android/blob/main/Documentation/guides/tracing.md

I'll do this myself when I get a chance to look at this, but that might point out specifically where the problem is.

[david-maw]

@janseris no, I wasn't actually trying to benchmark, I was trying to debug some code which calls SixLabors.ImageSharp, I did give it a try with the additions suggested by @jonathanpeppers though, alas release builds do not seem to be working for me. This one fails with

1>C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\32.0.415\targets\Microsoft.Android.Sdk.Aot.targets(78,5): error : Mono Ahead of Time compiler - compiling assembly C:\Users\david\Documents\Develop\Samples\ResizeImageMaui\ResizeImageMaui\obj\Release\net6.0-android\android-arm\aot-in\Mono.Android.dll
1>C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\32.0.415\targets\Microsoft.Android.Sdk.Aot.targets(78,5): error : AOTID AE96AD80-0388-6F60-8F1D-297A5B9027D9
1>C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\32.0.415\targets\Microsoft.Android.Sdk.Aot.targets(78,5): error : Compiled: 155736/155736
1>C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\32.0.415\targets\Microsoft.Android.Sdk.Aot.targets(78,5): error : Executing the native assembler: "C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\32.0.415\tools\binutils\arm-linux-androideabi-as"   -mfpu=vfp3 -o obj\Release\net6.0-android\android-arm\aot\armeabi-v7a\Mono.Android\temp.s.o obj\Release\net6.0-android\android-arm\aot\armeabi-v7a\Mono.Android\temp.s
1>Done building project "ResizeImageMaui.csproj" -- FAILED.

Not much of a problem for me actually, I'm mostly interested in debug builds, I don't move to a release build until the pre-release performance checks.

Just as well @NonameMissingNo tested it, thanks for that.

[jonathanpeppers]

@david-maw if a general Release build fails, you might file an issue here with a diagnostic MSBuild log:

https://github.com/xamarin/xamarin-android/issues

Thanks!

[NonameMissingNo]

@jonathanpeppers It would seem like that the ImageSharp is the one doing the work

maui-app_20220616_184909.speedscope.zip

[jonathanpeppers]

Thanks!

So I wonder if the code here:

https://github.com/SixLabors/ImageSharp/blob/7bd0e03792d2b5141fcfb046cb92329e9c0df582/src/ImageSharp/Processing/Processors/Transforms/Resize/ResizeWorker.cs#L168-L199

I bet there are some runtime optimizations for Span/Vector4 that we have on Windows + CoreClr, and we don't have those in Mono.

[david-maw]

@jonathanpeppers

Thanks for the hint on the diagnostic MSBuild Log. I'll look into the Release build issue a bit more and file a bug if it does not seem to be a local problem. The issue was building the MAUI Android version, not Xamarin.Forms Android. The error was in the AOT compiler which is presumably why the Debug build was ok.

Back to the original topic,

The Xamarin.Forms Android Release Build completed the second and subsequent image conversions in under 500 ms and a Debug build took 1.5 s, both a far cry from the MAUI Android numbers (just over 33 s for Debug, I can't test Release).

If the Android performance differential is a mono limitation why is the performance of the same app in Xamarin so much better? Is it that it is a different Mono version?

[jonathanpeppers]

Debug builds also default to UseInterpreter=true (this enables hot reload). This isn't an option at all in Xamarin.

So you could turn it off in your project, but then hot reload wouldn't work either.

[david-maw]

Thanks, that would probably work for the occasional test but so far at least it looks easier said than done, since for now isn't recognized (I note there are a number of Issues relating to handling it for multi-targeted project files.

[jonathanpeppers]

I had a conversation with some of the Mono folks -- going to move this to dotnet/runtime.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jonathanpeppers]

/cc @steveisok @lambdageek

[lambdageek]

1. We should try running this on the net7 runtime packs. There's been arm64 intrinsics work in net7 in mono.
2. The code that @jonathanpeppers pointed out (https://github.com/SixLabors/ImageSharp/blob/7bd0e03792d2b5141fcfb046cb92329e9c0df582/src/ImageSharp/Processing/Processors/Transforms/Resize/ResizeWorker.cs#L168-L199) looks like it could be tested in a console app. that should make it easier to look at what the JIT is doing.

[mjsb212]

So in .net 8 this will be fixed? I switched to SkiSharp as others said & performance is 100x better...but I love imagesharp for many things so this is a bummer

[EvgenyMuryshkin]

Same issue.
Native MAUI Downsize works, but rotates image.
SkiaSharp works, but also rotates images.

ImageSharp is the only one that works and does not rotate, but slow :(
Hope for MAUI 8, migration journey has been terrible.

[hallatore]

@EvgenyMuryshkin Are you trying to load an image with rotation metadata?

Try load it with image instead of bitmap. That ensures correct rotation.

using var image = SKImage.FromEncodedData(imageStream);
return SKBitmap.FromImage(image);

[EvgenyMuryshkin]

@EvgenyMuryshkin Are you trying to load an image with rotation metadata?

Try load it with image instead of bitmap. That ensures correct rotation.

using var image = SKImage.FromEncodedData(imageStream);
return SKBitmap.FromImage(image);

Does not work on iPad, Android seems ok

[e012345678]

Is there any workaround for the slow performance in ImageSharp for MAUI as of now 2024?

[JimBobSquarePants]

You’re asking the wrong people. Ask the Maui team.

EDIT. Note to self. Never use GitHub on your mobile. I thought the previous comment was in the ImageSharp downstream issue.

[MichalStrehovsky]

You’re asking the wrong people. Ask the Maui team.

Mono team, is there a way to rule out this is not (or confirm it is) caused by gsharedvt? Are there any events generated when gsharedvt code is hot? ImageSharp is generic heavy, it also uses generic virtual methods. Notice the hot stacks have methods instantiated over structs like Rgba32. I could easily see the perf being atrocious if every operation on a pixel changes from "load 32 bits into a register" to "memcpy a statically unknown number of bytes from X to Y". AFAIK Mono AOT lacks the analysis to figure out all necessary generic virtual method bodies and we often end up running gsharedvt versions.

[vargaz]

gsharedvt is only used on ios, not on android.

[Redth]

@SamMonoRT @steveisok any updates / progress in this area? It would be great to have ImageSharp running well on MAUI platforms and WASM!

[vargaz]

Does this really affect wasm ? I tried:

        for (int i = 0; i < 10; ++i) {
            var fs = typeof (Test).Assembly.GetManifestResourceStream ("Wasm.Console.V8.Sample.foo.jpg");
            var image = SixLabors.ImageSharp.Image.Load(fs);
        }

And it runs in about 2s on wasm in AOT mode on a 2000x1500 image, so each decode takes about 0.2s.

[vargaz]

The wasm AOT performance can probably be improved by pre seeding more generics, some methods are still interpreted because the AOT compiler doesn't know about them, like:
void SixLabors.ImageSharp.Formats.Jpeg.Components.Decoder.JpegImagePostProcessor:ConvertColorsInto<SixLabors.ImageSharp.PixelFormats.Rgba32> (SixLabors.ImageSharp.ImageFrame`1<SixLabors.ImageSharp.PixelFormats.Rgba32>)

[JimBobSquarePants]

How did you determine which method required pre seeding? We’ve added a lot of method seeding to the library already but it’s been without guidance.

What are the plans going forward to remove the requirement for pre seeding? It feels like a horrible hacky workaround. I would expect compiler analysis to do much better.

[vargaz]

The AOT compiler does appear to have problems figuring out which instances to generate. In this specific case, the caller is
ImageDecoderUtilities:Decode<Rgba32> which calls IImageDecoderInternals::Decode on an argument. So in theory, the aot compiler could figure out that the call could possible go to JpegDecoderCore::Decode<Rgba32> and generate that instance. Currently, this kind of analysis is not done.

The ImageSharp codebases unfortunately makes heavy use of generics, interfaces, valuetype generic arguments, etc., which is not very friendly to static compilation.

[JimBobSquarePants]

I can see that this has been removed from a performance backlog and nobody is now assigned. What can be done to change this state?

[sharpwood]

This is a serious issue: ImageSharp cannot be used at all on MAUI Android.

[MichalStrehovsky]

I can see that this has been removed from a performance backlog and nobody is now assigned. What can be done to change this state?

Cc @vitek-karas

[JimBobSquarePants]

Since there's been no movement here, I've created a potential workaround for this particular issue in the ImageSharp codebase, but I have no means to test my changes. Would anyone be able to compile and test the code in this PR

SixLabors/ImageSharp#2762

[JimBobSquarePants]

The sample projects are gone!! 😔

[JimBobSquarePants]

I'd like to note. @beeradmoore has been helping me investigate Android performance issues with MAUI and we've measured that release mode with LLVM performs adequately with the following configuration.

<PropertyGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android' AND '$(Configuration)' == 'Release'">
    <EnableLLVM>true</EnableLLVM>
    <RunAOTCompilation>true</RunAOTCompilation>
    <AndroidEnableProfiledAot>false</AndroidEnableProfiledAot>
</PropertyGroup>

However, we saw the significant performance issues highlighted in this issue running Android in Debug. I'm unsure whether removing the Release configuration condiftion here allows the library to build and run correctly. Guidance here would be appreciated.

Relevant discussion here:

SixLabors/ImageSharp#2762

[jonathanpeppers]

@JimBobSquarePants the configuration above looks correct to me for Release-mode for ImageSharp usage: "AOT everything" + LLVM.

In Debug-mode, we have the interpreter enabled by default, which probably doesn't play nice with hardware intrinsics / vector-based math. Does UseIntepreter=false make things any better? This will, of course, prevent C# hot reload from working.

[JimBobSquarePants]

@JimBobSquarePants the configuration above looks correct to me for Release-mode for ImageSharp usage: "AOT everything" + LLVM.

In Debug-mode, we have the interpreter enabled by default, which probably doesn't play nice with hardware intrinsics / vector-based math. Does UseIntepreter=false make things any better? This will, of course, prevent C# hot reload from working.

Thanks @jonathanpeppers I don't actually have the tooling to check but this sample application can be used to test against https://github.com/beeradmoore/ImageSharpMAUITest/tree/main

[beeradmoore]

I added this which I changed from true to false to confirm it was being applied.

<PropertyGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android' AND '$(Configuration)' == 'Debug'">
    <UseInterpreter>true</UseInterpreter>
</PropertyGroup>

Test run	JpgLoad	JpgResize	PngLoad	PngResize
UseInterpreter=true	14289.9ms	18556.9ms	123.4ms	344.7ms
UseIntepreter=false	14285.8ms	18491.2ms	122.9ms	349.3ms
UseInterpreter=false	1157.9ms	1387.3ms	35.4ms	42.2ms

What's interesting (and I guess explains it) is in my local source I still had this line enabled

 Console.WriteLine("Vector.IsHardwareAccelerated: " + (System.Numerics.Vector.IsHardwareAccelerated ? "True" : "False"));

This was true for my last set of release mode tests. I just checked and both of these debug modes

Vector.IsHardwareAccelerated: False

With UseInterpreter=true Vector.IsHardwareAccelerated is false, but with UseInterpreter=false Vector.IsHardwareAccelerated is true.

EDIT: This comment was updated to fix the typo UseIntepreter to UseInterpreter and then tests were re-run.

[JimBobSquarePants]

Why would intrinsics be turned off for debug mode? Not-only is that a performance concern but it also vastly transforms the profile of the running code, limiting the ability to identify codegen issues.

[jonathanpeppers]

@beeradmoore did I misspell above UseIntepreter? Sorry. Should be UseInterpreter?

Someone on the Mono team can comment, but it looks like hardware intrinsics for Vector is not implemented for the interpreter. It may be implemented for JIT on Android, if UseInterpreter=false (spelled correctly) makes things faster.

[beeradmoore]

Comment updated and tests re-run with correct spelling. Performance is a lot better, as expected Vector.IsHardwareAccelerated is also reporting true.

Aside from hot reload, is there anything else to consider when using UseInterpreter=false?