Slow/exploding build times with new workload version 8078

[DDHSchmidt]

The build-pipeline for the iOS version of our MAUI app unexplicably exploded in terms of build times and output size this tuesday.
What took around 20 minutes on monday now takes 70-80 minutes and the outputted ipa grew from 60 MB to ca. 90 MB.
There were no changes/checkins to (NuGet)dependencies, *.csproj, *.props, etc. files that could/should affect the build process inbetween these dates. It's virtually the same solution now taking three times longer.
On closer inspection of the build logs, the most prominent changes seemed to be in the automatically installed workloads by the command dotnet workload install ios maui --source https://api.nuget.org/v3/index.json

Monday:

Installing workload manifest microsoft.net.sdk.android version 34.0.79...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8022...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8022...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8022...
Installing workload manifest microsoft.net.sdk.maui version 8.0.61...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8022...
Installing workload manifest microsoft.net.workload.mono.toolchain.net7 version 8.0.0...
Installing workload manifest microsoft.net.workload.mono.toolchain.current version 8.0.0...
Installing workload manifest microsoft.net.workload.emscripten.net7 version 8.0.0...
Installing workload manifest microsoft.net.workload.emscripten.current version 8.0.0...

Tuesday:

Installing workload manifest microsoft.net.sdk.android version 34.0.113...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8078...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8078...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8078...
Installing workload manifest microsoft.net.sdk.maui version 8.0.61...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8078...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.2...

Adding an additional --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.61.json to the workload install command reverts to Monday's log output and decreases build time & output.

I'm sorry if I missed any announcements where new, drastical AOT changes were discussed, but the recent release notes only mention a "dedup optimization" which doesn't sound like something that should cause this situation, right?
Also: The linked PR mentions "FullAOT" mode, which we don't use. the relevant csproj-lines from our release-configuration are:

<MtouchExtraArgs>--optimize=-remove-dynamic-registrar</MtouchExtraArgs>
<MtouchEnableSGenConc>true</MtouchEnableSGenConc>
<MtouchInterpreter>-all,System.Private.Xml,System.Collections.Immutable</MtouchInterpreter>
<!-- System.Private.Xml is for our XSLT transformations that depend on reflection operations happening in that assembly -->
<!-- System.Collections.Immutable was chosen for an exception in CommunityToolKit's "toast" component: https://github.com/CommunityToolkit/Maui/Issues/1752 -->
<MtouchLink>SdkOnly</MtouchLink>

Steps to Reproduce

1. Build a Maui project for iOS via CI (Sorry, can't really share our project here :/ )

Expected Behavior

The build takes a "reasonable" amount of time to finish

Actual Behavior

The build takes 3 times longer for no discernable reason

Environment

Runner Image
Image: macos-13
Version: 20240707.2
Included Software: https://github.com/actions/runner-images/blob/macos-13/20240707.2/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20240707.2
Current image version: '20240707.2'

Build Logs

msbuild_workload_17.2.8022_redacted.binlog.zip

msbuild_workload_17.2.8078_redacted.binlog.zip

Example Project (If Possible)

Sorry, not possible I'm afraid, but I will reveal as much config/code you need, that I can.

[ivanpovazan]

As discussed over Discord, the regression comes from #20687

With deduplication disabled, the size regression is expected. However, we should investigate if we can do something about long build times when this optimization is disabled.

/cc: @kotlarmilos

[AStrickland-abbott]

I too see issues with this new workload version. IPAs generated go up 27MB in size and crash when launched. Our prior version was 17.2.8053 and we were running 8.0.100 dotnet. This is for our pipeline builds using dotnet publish.

[AStrickland-abbott]

I'm more concerned that workload versions can change when the .net version does not. This creates pipeline build inconsistencies if workload manifest can be changed on a whim for existing .NET versions

[DDHSchmidt]

I'm more concerned that workload versions can change when the .net version does not. This creates pipeline build inconsistencies if workload manifest can be changed on a whim for existing .NET versions

As a result of the current behaviour, we've gone and updated our current build script, namely the "dotnet workload install " step:
dotnet workload install ios maui --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.61.json --source https://api.nuget.org/v3/index.json

The --from-rollback-file parameter is new and should guarantee that the silent workload updates won't be pulled in by our automated builds.

[rolfbjarne]

I'm more concerned that workload versions can change when the .net version does not. This creates pipeline build inconsistencies if workload manifest can be changed on a whim for existing .NET versions

That's how workloads were designed: to keep you up-to-date with the latest version. As other comments have said, there's a way to install a specific workload version using a rollback file (see https://github.com/dotnet/sdk/blob/main/documentation/general/workloads/workload-rollback.md for more info).

[rolfbjarne]

IPAs generated go up 27MB in size and crash when launched

Please file a new issue for the crash so that we can investigate and try to find out happened.

[IainS1986]

Interesting! I've just ran into an app start crash this week/end of last week. I thought it was linked to a cert in our apple developer account (one just happened to expire this week but I didn't think it was related).

Might have ran into the same issue so will be try a workload rollback tomorrow and see if that 'fixes' the issue. Builds last week started crashing, the week before was fine.

We've had huge AOT times (and near 50% app size increase) since porting from Xamarin to .net around March time I think - I might try the deduplication fix on discord too and see if that resolves our large app size + long aot time too

[IainS1986]

Can't see a seperate issue for crashing so I'm going to raise one now (feel free to delete if there is one and I just couldn't see it).