AssemblyLoadContext issue with NuGet in Arcade Sleet publishing

[rainersigwald]

After #4916, something is causing a break dotnet/arcade#1965.

[rainersigwald]

@chcosta used an instrumented MSBuild which helped isolate the problem: the Microsoft.DotNet.Build.Tasks.Feed package contains:

• SleetLib, which references
  • NuGet at package version 4.3.0
• NuGet at package version 5.3.0

The loader currently requires strict version matches

https://github.com/microsoft/msbuild/blob/8aa0b87c00c6f26a565cf5e10975769dad9f378b/src/Shared/MSBuildLoadContext.cs#L71-L75

So when we try to load NuGet-as-referenced-from-Sleet, it fails because it can only find NuGet 5.3.0.4 and Sleet references 4.3.0.5.

This wasn't a problem pre-ALC isolation, because we always loaded NuGet from the sdk root folder next to MSBuild. Now that's just a fallback, and the assemblies in the package match by simple name.

What I don't yet understand is what happens after we try this and it fails. Why don't we then fall back to next-to-MSBuild?

[rainersigwald]

What I don't yet understand is what happens after we try this and it fails. Why don't we then fall back to next-to-MSBuild?

I strongly suspect that this will be fixed by #6558.

@chcosta do you remember this issue? Could we try backing dotnet/arcade@841c83d out in an experimental way?

[chcosta]

I very much remember this issue and am excited to get this resolved. We can get a private build of Arcade published with that change removed. I should be able to help out tomorrow.

[rainersigwald]

@chcosta This is in the latest preview release checked in now so we can probably go ahead with an Arcade fixup after the next SDK preview release.

[chcosta]

I'm not sure I'll have time to look at this, this week. @dnceng, may be able to help out.

@dotnet/dnceng , this requires undoing the hack in the feed package, and validating that doesn't break the sdk /runtime repos before committing.

[greenEkatherine]

@chcosta could you please point out an issue or PR related to that hack? I would like to help but I don't have a context

[chcosta]

The hack is mentioned above as dotnet/arcade@841c83d. I'm on my phone at the moment, but I think that's the right one

[chcosta]

@rainersigwald , What version of the sdk contains the fix?

[chcosta]

This appears to still be failing with sdk "6.0.100-rc.1.21379.2" and MSBuild "17.0.0-preview-21416-02"

[rainersigwald]

Hmm, that MSBuild version should have #6558. Guess that didn't fix it then (though I was pretty confident it would and don't understand how it didn't . . .).

I marked this to pull it into our current list of things to investigate.

[chcosta]

I couldn't actually get a repro using a current Arcade. I think the code path for publishing has changed significantly and we no longer trip this. However, using the old Arcade with the new SDK / MSBuild, caused the same error we previously saw. I can try to get this under a debugger again if needed.

[marcpopMSFT]

Is the old Arcade still in wide usage by teams and are those teams using the new SDK/MSBuild? That seems like a fairly unlikely combo. Seems like we should just close this as we unblocked the one customer we got a report from. Any concerns?

[MattGal]

Is the old Arcade still in wide usage by teams and are those teams using the new SDK/MSBuild? That seems like a fairly unlikely combo. Seems like we should just close this as we unblocked the one customer we got a report from. Any concerns?

With the end-of-life of 2.1 (except perhaps certain custom builds) I think our actual usage of sleet is effectively over, so I'd be fine closing this as long as @chcosta agrees.

My only concern, and I think we can address this if it happens, is that at some point we may not have old-enough desktop msbuild on agents to not hit the issue and this could result in some fixups to the (dead) builds.

[chcosta]

I'm completely ok with closing this. We have a workaround in Arcade, and as far as I can tell, this is not actually affecting Arcade anymore.