Use EnterContextualReflection to address .NET Core dependency loading issues

[ChrisMaddock]

Hopefully fixes #828 and #916

This uses the new EnterContextualReflection API introduced in netcore30, which will hopefully solve our .NET Core dependency loading issues. I wasn't aware of this previously as I don't think Microsoft's docs have yet been updated to include this - however there's some design documentation at the link below. (Thanks @Learfin for mentioning it in your write-up!) From what I understand, the API was added to tackle issues with xUnit that seem to be very similar to the problems we've been hitting.

https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/AssemblyLoadContext.ContextualReflection.md

I wanted to add a test for this, but unfortunately it's not really possible our current build structure. It's another example where a proper suite of acceptance test assembly's would be really handy - hopefully we'll get round to that one day.

Note the only change is in the final commit - this is just branched off #923 as they will interact. Once #923 is merged, I'll rebase this.

---

@Learfin, @joeltankam, @VoltanFr, @bouchraRekhadda, @runehalfdan, @webman1 - I gather you've all encountered this issue in one form or another. I've tested against the repro's provided in #916 and #939, but would really appreciate anyone who could test against their particular usecase before merging this. You should be able to download updated packages from the CI build here: https://ci.appveyor.com/project/CharliePoole/nunit-console/builds/38972209/artifacts

[ChrisMaddock]

CI is very confusing. Looks like Travis is failing due to the changes in #835 - and for some reason has just started running again, after not running for the last 7 months?

No idea what's caused that. I think the issue is just that net50 and netcoreapp21 aren't installed. I think this is probably our cue to remove Travis, given we're now maintaining 3 separate CI systems. As far as I'm aware, everything tested on Travis is also covered by Azure.

[CharliePoole]

@ChrisMaddock How come this PR is including my changes on the other PR, especially since you've asked me to make changes before committing?

[CharliePoole]

OK, never mind. I see you have a comment about that in the issue.

[bouchraRekhadda]

Hello @ChrisMaddock, thank you for the fix.
I've tried to run tests (similar to ones described in #939 ) through the package 3.13.0 you have provided us, unfortunately it crashed before launching any test:

NUnit.Engine.NUnitEngineException : The NUnit 3 driver encountered an error while executing reflected code.
----> System.InvalidCastException : Unable to cast transparent proxy to type 'System.Web.UI.ICallbackEventHandler'.
--NUnitEngineException
The NUnit 3 driver encountered an error while executing reflected code.
Server stack trace:
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
--
InvalidCastException
Unable to cast transparent proxy to type 'System.Web.UI.ICallbackEventHandler'.
at NUnit.Framework.Api.FrameworkController.LoadTestsAction..ctor(FrameworkController controller, Object handler) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 504
Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0

One last thing, I've noticed that it claims the environment is .NET Framework instead of .NET Core 3.1

As opposed to Nunit.ConsoleRunner.NetCore 3.12.0-beta2:
.

[ChrisMaddock]

Thanks for trying it out @bouchraRekhadda! Just to check, that's definitely with the NUnit.ConsoleRunner.NetCore package, and not the normal one?

Could you come up with a minimal example that reproduces this, please? Unfortunately there's not enough info in the stack trace to work out what's going wrong.

[bouchraRekhadda]

Hello @ChrisMaddock,

Could you come up with a minimal example that reproduces this, please? Unfortunately there's not enough info in the stack trace to work out what's going wrong

I'll try to provide tomorrow a reproduction code for binary serialization particular case.

Regarding other use cases with assembly load issues, we have one more where we use code compilation at runtime. Please find more details in nunit-sample repository with a little description of how to reproduce in the README (I hope this will be helpful).

[runehalfdan]

I haven't managed to catch up with all issues that is fixed in this version of the console runner, so could be I have tested things that is known not to work

But I downloaded it and gave it a run. Had to add <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> to all test-projects to avoid FileNotFoundException:
System.IO.FileNotFoundException : Could not load file or assembly 'nunit.framework, Version=3.13.2.0,

Also still cannot run tests out-of-process. (Since we build all our test-assemblies to separate outputs, some shared dlls need to have some static data initialized, and the runtime gets confused about this; first test-assembly works, the seconds don't initialize the static, but don't find the data either)

Also, for Bamboo reasons, we need to have NUnit2 output.
And Nunit project files would be nice, but any methods that allows use to list all test-assemblies in a file is ok.

Edit: We are target .Net Core 2.2

[bouchraRekhadda]

Hello @ChrisMaddock

Regarding the above stack-trace, and after pulling my hair out, I finally understood why it occurred: we have disabled the Target Framework attribute generation through <GenerateTargetFrameworkAttribute>False</GenerateTargetFrameworkAttribute>, which I believe confused NUnit.Console into thinking it is a .NET Framework assembly instead of .NET Core.

So after enabling the attribute generation, I can finally confirm that the current version 3.13.0 fixes the serialization issue we had (cf. #939 ) 🎉.

[CharliePoole]

Wow! Just read this and realized we have another way we should test the engine... disabling generation of TargetFrameworkAttribute! Thanks for that @bouchraRekhadda.

Yes, The engine currently assumes the current framework. But the change you are testing was based on an earlier version of my unmerged PR #937 which changes that to assume .NET Framework, since I've only ever seen a missing attribute for netfx.

Really, we shouldn't assume anything but when the attribute is not there, we should go further in trying to figure out what the assembly actually targets. 😉

[bouchraRekhadda]

Regarding other use cases with assembly load issues, we have one more where we use code compilation at runtime. Please find more details in nunit-sample repository with a little description of how to reproduce in the README (I hope this will be helpful).

Regarding this scenario (code emission at runtime), unfortunately I still reproduce the failures with 3.13.0.
In UnitTests.EmitDemo I basically compile a VB code that's referencing a .NET Core 3.1 library project class, I do an Assembly.LoadFrom() that ends up adding my compiled VB assembly in the AssemblyLoadContext.Default, when I try to get assembly types it returns null.

@ChrisMaddock, @CharliePoole what do you advise for this case ?

[ChrisMaddock]

@bouchraRekhadda Great news we've now got the normal case up and running! 🎉 Your compilation at run-time case, afraid I have no idea how to get that working. 😅 I would have thought EnterContextualReflection() would have encapsulated something like that - but it sounds like it doesn't. I think what we have here is an improvement, so lets merge this and move the new case to a new issue - would appreciate any ideas you or your team have to get that example working too!

---

@runehalfdan Hi - good to hear from you again! 👋

Please could you elaborate on "Had to add true to all test-projects"? It would be really helpful if you could add a small repo for this problem (in a new issue, perhaps!). As things stand, we don't currently have any more reproducible examples which are not working, other than @bouchraRekhadda's code-compliation-at-runtime, which I expect is a very different problem!

On the other points you mention:

• Out of process running is now that bit closer, thanks to Charlie's work in Run .NET Core Tests from NUnit3 Console #937.
• nunit2 and project files I believe are just awaiting an interested contributor to convert the extensions. Although @CharliePoole may know more about where those are at - he currently maintains those extension.

---

@CharliePoole - What are the chances that the first user to try this would encounter the TargetFrameworkAttribute issue we'd just been talking about, eh? 😄

[ChrisMaddock]

I'm going to merge this, as it seems to be at least one step closer to where we want to be. Thanks to those who've been able to try it out. 🙌

[runehalfdan]

@runehalfdan Hi - good to hear from you again! 👋

Please could you elaborate on "Had to add true to all test-projects"? It would be really helpful if you could add a small repo for this problem (in a new issue, perhaps!). As things stand, we don't currently have any more reproducible examples which are not working, other than @bouchraRekhadda's code-compliation-at-runtime, which I expect is a very different problem!

On the other points you mention:

• Out of process running is now that bit closer, thanks to Charlie's work in Run .NET Core Tests from NUnit3 Console #937.

• nunit2 and project files I believe are just awaiting an interested contributor to convert the extensions. Although @CharliePoole may know more about where those are at - he currently maintains those extension.

Edited original;markdown problems. Was CopyLocalLockFileAssemblies that must be true.

[bouchraRekhadda]

Hello @ChrisMaddock, first of all thank you for the fix, I can't wait to update our Test Runner with this new version.

Regarding this:

Your compilation at run-time case, afraid I have no idea how to get that working. 😅 I would have thought EnterContextualReflection() would have encapsulated something like that - but it sounds like it doesn't. I think what we have here is an improvement, so lets merge this and move the new case to a new issue - would appreciate any ideas you or your team have to get that example working too!

Always using version 3.13.0 provided through this pull request, we have tried enforcing the current reflection context by simply doing
var assembly = (AssemblyLoadContext.CurrentContextualReflectionContext ?? AssemblyLoadContext.Default).LoadFromAssemblyPath(assemblyPath); instead of hard-coding the NUnit ALC, and it worked 🙆‍♀️ (the change is available here).

[bouchraRekhadda]

Hello, @ChrisMaddock, @CharliePoole,
Do you know when will 3.13.0 be publicly released ? we could use a beta as well if you think the official release will take time.

Thanks

[ChrisMaddock]

Hi @bouchraRekhadda - probably not imminently I'm afraid, there's a few more issue we need to look into first like #956 and #843. You can get pre-release builds from our MyGet feed though, that should cover what you need? 🙂

https://www.myget.org/feed/nunit/package/nuget/NUnit.ConsoleRunner.NetCore

[bouchraRekhadda]

You can get pre-release builds from our MyGet feed though, that should cover what you need?

We were hoping to get a beta release like 3.12.0.beta2.

To be more precise, we only need NUnit.Engine including the fix from this pull request; we could re-package internally the nunit.engine.dll from NUnit.ConsoleRunner.NetCore 3.13.0-dev-05299 and use it instead in our custom Test Runner, but we prefer to avoid this kind of hacks if possible.

[bouchraRekhadda]

Hello,
Any update on my request above please ?

Thanks