Run .NET Core Tests from NUnit3 Console #937

CharliePoole · 2021-04-13T16:00:13Z

Another step toward issue #923.

This PR adds TCP as a transport and uses it to run tests under .NET Core.

A .NET Core 3.1 agent has been added. We'll have to decide whether we want others as separate agents. A single package test has been added to exercise the agent. It is run for the NuGet, Chocolatey and Zip packages of the console runner.

The MSI package does not yet include the .NET Core agent. I made a start on this but backed out the changes when I saw that we need to make some decisions about the msi package. I think this will be easier to do if there's a separate PR for it. This one is pretty big as it is.

CharliePoole · 2021-04-13T18:00:13Z

Unable to figure out why the Azure builds are failing.

Windows job can't find src\NUnitEngine\nunit-agent\obj\Release\x86\netcoreapp3.1\apphost.exe, which is present when I do a local build.

Linux and MacOs jobs say that the net-4.0 framework isn't available.

Not being an Azure guy, I don't even know where to look.

src/NUnitEngine/nunit.engine.core/RuntimeFramework.cs

ChrisMaddock · 2021-04-13T21:31:01Z

Not sure if the CI has been re-run since your comment Charlie, but looks like the Windows build is ok. Looks to me like the Mono tests are all failing to start up an agent on Linux/OSX. Not sure what's in this PR that would affect that, however...

CharliePoole · 2021-04-15T14:31:46Z

@ChrisMaddock @jnm2

The error in building windows continues to appear intermitently, i.e. failure to locate apphost.exe, which should be generated by the restore. I have noticed that when it does appear, there is nothing in the log about restoring nunit-agent.csproj. OTOH nunit-agent-x86.csproj is restored correctly.

I'm working on the hypothesis that nuget is choking on the odd setup we have for the two agent projects: two projects in one directory with separate intermediate output folders, obj and obj\x86. We always knew this was odd and it was somewhat surprising that it actually worked. But I suspect it isn't working for .NET Core builds.

I'll run an experiment, separating the two projects into two different folders and replicating the code.

CharliePoole · 2021-04-15T16:19:31Z

That worked! The latest commit builds under Windows on both AppVeyor and Azure. The Linux and MacOs builds are still failing on Azure, but not from the same cause.

CharliePoole · 2021-04-30T06:08:36Z

The result is that AppVeyor doesn't seem to have the X86 version of dotnet.exe installed. We could probably install it in our script, but I'd rather save that for later. Instead I make a check for it's availability before trying to run the test, which is more robust.

After this last commit, the only remaining issue is the msi build.

CharliePoole · 2021-05-01T05:03:03Z

@ChrisMaddock OK, last bit is done and this is ready for final review.

ChrisMaddock

Great to see this all coming together Charlie! Just a few niggles at this point. 🙂

Just checkpointing the review here as GitHub's gone a bit funny - still a bit more to go through...

ChrisMaddock · 2021-05-02T14:02:54Z

src/NUnitConsole/nunit3-console.tests/App.config

+  <appSettings>
+    <add key="test.setting" value="54321"/>
+  </appSettings>
+</configuration>


This file is included within the "add agent to msi" commit - is it included accidentally? I'm not sure what the purpose of this is.

I had to add it to make all tests pass. There's a test that reads it. In fact, this should eventually be cleaned up and we should stop using .config files entirely for .NET Core as a part of reworking RuntimeFramework. .runtimeconfig.json is where this should end up.

Hard to figure out the boundaries of a PR... i.e. where do you stop and create a new issue. Personally, I'd prefer not to deal with this one while we still allow --inprocess because that makes RuntimeFramework more complicated. I do think that RuntimeFramework is a good place for me to exercise my preference for tiny PRs. 😃

src/NUnitEngine/nunit.engine.tests/nunit.engine.tests.csproj

ChrisMaddock · 2021-05-02T14:11:25Z

msi/nunit/engine-files.wxi

+                      ProcessorArchitecture="msil"
+                      Source="$(var.InstallImage)bin/agents/netcoreapp3.1/nunit-agent.dll" />
+                <File Id="nunit_agent.runtimeconfig.json"
+                      Source="$(var.InstallImage)bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" />


I'm not totally clear on what is and isn't required for .NET Core, but I think this should also include nunit-agent.deps.json.

I was also wondering about nunit-agent.dll.config, but I think is actually netfx specific, so it should probably be removed entirely from the netcoreapp agent? I'm not sure how to do that off the top of my head, will give it a go.

I believe that nunit-agent.deps.json is only for use in building the project. It causes nunit-agent.runtimeconfig.json to be generated. If that's wrong, I can add it. Do you have different info? At this point all the tests pass using all the packages.

Hmm, ok. I always thought it should be as it details where to load dependencies from at runtime. I'm struggling to find any formal documentation however, or any clear answer elsewhere.

This page seems to be the closest there is. From my reading of that, for a framework-dependent deployment as ours is, the file should be included. What do you think?

https://github.com/dotnet/cli/blob/master/Documentation/specs/runtime-configuration-file.md

ChrisMaddock · 2021-05-02T14:18:44Z

msi/nunit/engine-files.wxi

+            <Component Id="NUNIT_AGENT_ADDINS_NETCORE31" Location="local" Guid="21D2D9FB-A57D-4B02-92A0-6697674FB869">
+                <File Id="nunit.agent.netcore31.addins" Source="$(var.InstallImage)nunit.agent.addins" />
+            </Component>


Suggested change

<Component Id="NUNIT_AGENT_ADDINS_NETCORE31" Location="local" Guid="21D2D9FB-A57D-4B02-92A0-6697674FB869">

<File Id="nunit.agent.netcore31.addins" Source="$(var.InstallImage)nunit.agent.addins" />

</Component>

I think this addins file points to a dir of netfx extensions, none of which can be loaded by the netcoreapp31 agent. Shall we just leave this out for now? We might want to package netcoreapp-compatible extensions with the engine in future, but they'll probably need to be in a different subdir anyway.

Good point. I'll take a look. In fact, this directory should only used for the .NET Core 3.1 agent, so I may be able to just change the content of either the addins file or the directory itself.

ChrisMaddock · 2021-05-02T14:20:40Z

nuget/engine/nunit.engine.nuspec

+
+    <file src="bin/agents/netcoreapp3.1/nunit-agent.dll" target="contentFiles/any/agents/netcoreapp3.1" />
+    <file src="bin/agents/netcoreapp3.1/nunit-agent.dll.config" target="contentFiles/any/agents/netcoreapp3.1" />
+    <file src="bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" target="tools/agents/netcoreapp3.1" />


Suggested change

<file src="bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" target="tools/agents/netcoreapp3.1" />

<file src="bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" target="contentFiles/agents/netcoreapp3.1" />

I think this one's been missed, Charlie. (I also don't know why we use contentFiles instead of tools here, but keeping it consistent seems safest! 😅)

I initially tried to keep it consistent, but I got failures when I did that. However, I can now think of another reason for that, so I'll try again. If this were for 4.0, I'd say just drop the whole contentFiles thing unless somebody can say why.

src/NUnitEngine/mock-assembly-x86/mock-assembly-x86.csproj

src/NUnitEngine/nunit-agent-x86/nunit-agent-x86.csproj

src/NUnitEngine/nunit.engine.core/nunit.engine.core.csproj

src/NUnitEngine/nunit.engine/Services/RuntimeFrameworkService.cs

ChrisMaddock · 2021-05-02T15:06:50Z

Ok - I think I've covered everything here.

I've resolved a few discussions that I think are sorted, and unresolved a couple that I think were missed. I've added a few new comments, almost entirely re: the new changes. I think I've also responded to all still-open conversations.

The x86 changes look good - glad finally we've got some tests around that too! In future, I'd like to add tests to mock-assembly which only pass if they are running on the targeted framework/architecture, so we can smoke test that functionality too.

Let me know if there's anything I've missed - let's get this merged! 😁

CharliePoole · 2021-05-02T15:52:09Z

I answered some questions and resolved a couple of items but I'll go through all the items again. It's really hard to track all this with so many changes, but I'm keeping a page open where I've marked all the ones I no longer want to see as "viewed". WRT testing that it really runs under the targeted runtime and architecture... I'd like to do that too, but when I took a casual look at the idea, it seemed as if I would have to add more info to TestResult.xml to do that. How about if I do whatever can be done without changing the XML format and then trying to do more when we incorporate this in the 4.0 branch?

…

On Sun, May 2, 2021 at 8:07 AM Chris Maddock ***@***.***> wrote: Ok - I *think* I've covered everything here. I've resolved a few discussions that I think are sorted, and unresolved a couple that I think were missed. I've added a few new comments, almost entirely re: the new changes. I think I've also responded to all still-open conversations. The x86 changes look good - glad finally we've got some tests around that too! In future, I'd like to add tests to mock-assembly which only pass if they are running on the targeted framework/architecture, so we can smoke test that functionality too. Let me know if there's anything I've missed - let's get this merged! 😁 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#937 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUN7BPDYUSNIJ3UWHEDWMLTLVTBPANCNFSM423UETAQ> .

ChrisMaddock · 2021-05-02T17:16:51Z

Thanks Charlie - and it is a lot to keep track of. Hopefully we'll catch most things between us!

I'd like to do that too, but when I took a casual look at the idea, it
seemed as if I would have to add more info to TestResult.xml to do that.

Ah - I was thinking more simply a test which fails when it's run on e.g. .NET 2.0, and using pre-compilation symbols to make sure the right version is included in the right assembly. That's what I do in my manual tests before releases. 🙂

CharliePoole · 2021-05-02T19:54:33Z

If I understand you, that would work if the test can run, e.g. net20 on net40. But if the test can't run at all, the package tests typically need to examine the TestResult file to find out why it failed. That's because we keep running subsequent tests rather than throwing immediately. I may be able to figure out a way to recognize that the test failed because the agent couldn't load it though.

CharliePoole · 2021-05-03T15:54:49Z

@ChrisMaddock This PR and reviews and comments on reviews etc. is getting a bit hard to follow. So to checkpoint where we are, here's a list of items you brought up, which I believe still need to be dealt with...

Figure out whether we need to package nunit-agent.deps.json. I have tests passing without it, but it won't hurt to add. We should either include it in all four packages or leave it out of all.
Decide about .addins files for agents. Given that we currently have no extensions of our own for .NET Core, we could just leave out the file. Or we could point to a directory that users can put things in... or... or... or... It would be nice if we didn't need to re-release the first time there's a .NET Core extension, although that's possible as well.
Decide whether to use contentFiles or tools for the .json files. My memory is that using contentFiles didn't work, but I'll retest that.
For mock-assembly, I'll test out your idea of just including the files in the parent dir without a link.
Check whether various references and package references are really needed for the tests to pass. I'm a little vague here because you hadn't followed up to some of my responses. Basically, however, I think we need to leave in any references if the tests start to fail when I remove them. It may be that this works differently if I include .deps.json, so I'll try with and without. That said, at some future point it would help if you could write something about why those package refs and refs ought to be removed. Maybe that should be a 4.0 issue.
Remove extra checks from RuntimeFrameworkService.FrameworksMatch
Re-examine default behavior in RuntimeFrameworkService.SelectRuntime when no RuntimeTargetAttribute is present.

Let me know if I've missed something. 😃

ChrisMaddock · 2021-05-03T16:59:51Z

That looks about right to me! 👍

I'm in favour of taking the easy way out for the msi, and just removing the file until we have extensions to go with it.
I did try myself this before suggesting, and it seemed to work for me. 🙂 (Same suggestion for the new nunit-agent projects, too!)
Can you highlight to me which responses I've missed? I think this covers two separate issues:
5a. The references added to nunit.engine.core.csproj, which I believe are just a rebase error as I'd just removed those from master recently.
5b. The reference added to nunit.engine.tests.csproj, and the new app.config added to the nunit-3console.tests project. Both of these - I'm unclear on why the tests have been passing previously (I think!), and why they're now needed. Would appreciate your thoughts there.

CharliePoole · 2021-05-03T18:19:53Z

@ChrisMaddock Regarding (5) I probably didn't make myself clear. It's not a rebase error. I saw that you had removed them and I had to put them back in to the package tests for .NET Core to pass. There was no failure before because we didn't have such a test or even a .NET Core agent to fail!

That said, I'm taking on trying again without them and seeing if this interacts with the presence or absence of `nunit-agent.deps.json'. I think this is something we really need to understand better.

CharliePoole · 2021-05-04T16:51:07Z

Should be ready to go now. I think we have a number of packaging decisions to make for 4.0, however.

ChrisMaddock · 2021-05-04T19:40:52Z

Great, thanks for getting those sorted! Afraid there's a couple of things in the last commit I still think need looking at.

The latest commit has added nunit-agent.runtimeconfig.dev.json to the packages, rather than nunit-agent.deps.json. I presume that was unintentional, as you'd put deps.json in your list above? The "dev" file I believe is for development only.
Sorry, we've miscommunicated around the .addins files. By my suggestion, I was only meaning removing it from the msi, as we know that there are no .NET Core compatible extension assemblies installed with the msi. I see the last commit also removes it from the NuGet packages. I think the NuGet version needs to stay I'm afraid, I know that's currently used at least for the TeamCity extension.

Let me know if there's anything we need to discuss further - otherwise please go ahead and merge once the changes are in. Thanks for all your work getting it sorted! 😄

CharliePoole · 2021-05-05T07:12:50Z

I'm afraid I thought I was doing what you wanted without really understanding why you wanted it. I'll take the blame for that choice. In future, I think it would help to have failing scenarios that force us to add something to the packages. After all, that's why we have package tests.

I'll make the changes in the morning.

ChrisMaddock · 2021-05-09T12:02:49Z

Great - thanks for this Charlie! One step closer to where we want to be! 😁

CharliePoole added 10 commits April 2, 2021 09:47

Rename folder to match namespace

8316c81

Descriptions of each EngineStatus value

e5d1cf8

Add TCP to engine core

f6b92a3

Add TCP Transport to nunit.engine

741d463

Connect TestAgency to both transports

08777aa

Build agent for .NET Core 3.1

f94d360

Modify process runner to use .NET Core 3.1 agent

6b108de

Update package content

381f1ae

Add nunit3-console package test under .NET Core 3.1

82ca51d

Fix error in MSI package tests

c8f6fc0

CharliePoole requested review from ChrisMaddock and jnm2 April 13, 2021 16:00

Merge with latest master

8759962

CharliePoole changed the title ~~Issue 923b~~ Run .NET Core Tests from NUnit3 Console Apr 13, 2021

CharliePoole added the awaiting:review label Apr 13, 2021

ChrisMaddock reviewed Apr 13, 2021

View reviewed changes

src/NUnitEngine/nunit.engine.core/RuntimeFramework.cs Show resolved Hide resolved

Remove 32-bit build of .net core agents

c3556d0

CharliePoole added awaiting:contributor and removed awaiting:review labels Apr 15, 2021

Move nunit-agent-x86 to a separate folder

a6d3e9e

CharliePoole force-pushed the issue-923b branch from 9e39007 to a6d3e9e Compare April 15, 2021 15:06

CharliePoole force-pushed the issue-923b branch 3 times, most recently from 0c8666d to d5d706e Compare April 15, 2021 21:46

Recognize that mono can run .Net FX assemblies

eda6812

CharliePoole force-pushed the issue-923b branch from d5d706e to eda6812 Compare April 16, 2021 01:00

Give error if x86 version of dotnet is needed but not available

3e3b43d

CharliePoole force-pushed the issue-923b branch from a24190f to 3e3b43d Compare April 30, 2021 04:34

Skip running X86 .NET Core tests where not installed

7963ff2

Add .NET Core 3.1 agent to msi

154866e

ChrisMaddock reviewed May 2, 2021

View reviewed changes

ChrisMaddock mentioned this pull request May 2, 2021

Allow netcoreapp agent to work with non-standard dotnet install locations #941

Closed

ChrisMaddock reviewed May 2, 2021

View reviewed changes

src/NUnitEngine/nunit.engine/Services/RuntimeFrameworkService.cs Outdated Show resolved Hide resolved

ChrisMaddock mentioned this pull request May 2, 2021

NUnit.Engine 3.12.0 .NET Core and TypeDescriptor.GetConverter([...]) dependency loading problems #916

Closed

ChrisMaddock mentioned this pull request May 2, 2021

Make default TestAgency URI unique per run #931

Closed

CharliePoole mentioned this pull request May 3, 2021

Multicomputer/Multiplatform testing environment #944

Open

Changes arising from last review

116d081

CharliePoole mentioned this pull request May 5, 2021

Use EnterContextualReflection to address .NET Core dependency loading issues #942

Merged

Final fixes

d81b3ba

ChrisMaddock merged commit 7822971 into master May 9, 2021

ChrisMaddock deleted the issue-923b branch May 9, 2021 12:02

This was referenced May 9, 2021

nunit.engine.tests are failing to unload on master branch #949

Closed

Dispose classes as required in Tests to prevent failed app domain unload #952

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run .NET Core Tests from NUnit3 Console #937

Run .NET Core Tests from NUnit3 Console #937

CharliePoole commented Apr 13, 2021

CharliePoole commented Apr 13, 2021

ChrisMaddock commented Apr 13, 2021

CharliePoole commented Apr 15, 2021

CharliePoole commented Apr 15, 2021

CharliePoole commented Apr 30, 2021

CharliePoole commented May 1, 2021

ChrisMaddock left a comment

ChrisMaddock May 2, 2021

CharliePoole May 2, 2021

ChrisMaddock May 2, 2021

CharliePoole May 2, 2021

ChrisMaddock May 2, 2021

ChrisMaddock May 2, 2021

CharliePoole May 2, 2021

ChrisMaddock May 2, 2021

CharliePoole May 2, 2021

ChrisMaddock commented May 2, 2021

CharliePoole commented May 2, 2021 via email

ChrisMaddock commented May 2, 2021

CharliePoole commented May 2, 2021

CharliePoole commented May 3, 2021 •

edited

Loading

ChrisMaddock commented May 3, 2021

CharliePoole commented May 3, 2021

CharliePoole commented May 4, 2021

ChrisMaddock commented May 4, 2021

CharliePoole commented May 5, 2021

ChrisMaddock commented May 9, 2021

	<Component Id="NUNIT_AGENT_ADDINS_NETCORE31" Location="local" Guid="21D2D9FB-A57D-4B02-92A0-6697674FB869">
	<File Id="nunit.agent.netcore31.addins" Source="$(var.InstallImage)nunit.agent.addins" />
	</Component>

	<file src="bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" target="tools/agents/netcoreapp3.1" />
	<file src="bin/agents/netcoreapp3.1/nunit-agent.runtimeconfig.json" target="contentFiles/agents/netcoreapp3.1" />

Run .NET Core Tests from NUnit3 Console #937

Run .NET Core Tests from NUnit3 Console #937

Conversation

CharliePoole commented Apr 13, 2021

CharliePoole commented Apr 13, 2021

ChrisMaddock commented Apr 13, 2021

CharliePoole commented Apr 15, 2021

CharliePoole commented Apr 15, 2021

CharliePoole commented Apr 30, 2021

CharliePoole commented May 1, 2021

ChrisMaddock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisMaddock commented May 2, 2021

CharliePoole commented May 2, 2021 via email

ChrisMaddock commented May 2, 2021

CharliePoole commented May 2, 2021

CharliePoole commented May 3, 2021 • edited Loading

ChrisMaddock commented May 3, 2021

CharliePoole commented May 3, 2021

CharliePoole commented May 4, 2021

ChrisMaddock commented May 4, 2021

CharliePoole commented May 5, 2021

ChrisMaddock commented May 9, 2021

CharliePoole commented May 3, 2021 •

edited

Loading