What shall we call the V4 console runner?

[CharliePoole]

Clearly, nunit3-console.exe no longer works. Could be nunit4-console.exe of course. Or something entirely different?

@nunit/engine-team What do you think?

[jnm2]

Having the number in there struck me as odd, but if the containing folders are in %path% (or invoked as .NET global tools) then this would allow you to have multiple at once and choose which one you want. I'm not sure if this is a likely situation.

What about nunit.exe? Most console apps don't have console in the file name.

[CharliePoole]

Nunit.exe was once the name of our gui, but of course, that's no longer relevant.

These days, I can see two possible drawbacks, but I'm not sure if they are really a problem...

1. We currently use dotnet nunit to run our CLI tool. If we keep the same design, where the net6.0 build of the console runner becomes the tool itself, then no problem. But if we have a separate tool, even just a wrapper, it might be a conflict.

2. I've toyed with the idea of a general nunit command, with subcommands. It may be worthwhile to keep that door open.

"Most console apps don't have console in the file name."

Good point! maybe we should look at what others call their batch runners.

One thought... we could use nunit4-console while we are developing to avoid confusion with version 3 and plan on a final selection before the actual release.

[OsirisTerje]

The name is also equal to the nuget id. Changing it would mean that the chain would be broken there. It might not be a problem for the console itself though, since it is a separate tool. The adapter and the engine which are included in projects though, it would be much harder to change. Calling it nunit.exe could be interesting, but it would also mark it as THE Nunit tool. Perhaps it is?

[CharliePoole]

@OsirisTerje

Actually, the nuget id for nuget.org is NUnit.ConsoleRunner while the one used for chocolatey.org is nunit-console-runner. The name of the exe included in the packages makes a breaking change, of course, but this is a major update so it's permitted.

Good point about the tool. Our two notes may have crossed. Currently the net 6.0 build of the console runner is the dotnet tool. However, it lacks some features a tool needs and we plan to add features a tool doesn't usually have. So I'm thinking that the tool (except for it's name) is up in the air.

All stated, maybe we have to leave it up in the air for a while. :-)

Charlie

BTW, Terje, I hope you are following the discussion of breaking changes in the engine!

[OsirisTerje]

I have been drowning in NUnit notifications, see there are a bunch I have missed. Thanks for the heads up on the breaking changes, I'll get to that discussion.

Another thing, I tried to install the dotnet core tool, but that fails. (Ref https://www.nuget.org/packages/NUnit.ConsoleRunner.NetCore/) But this might be the outdated one?? Also, the NUnit.Consolerunner is a package, not a tool (which doesn't make sense to me) . What am I missing here?

[CharliePoole]

NUnit.ConsoleRunner previously built two console runner targets: net20 and netcoreapp3.1. Using a lot of conditional code, the two builds really end up producing what amounts to two different applications. The netcoreapp3.1 is actually built as a tool. So you have two executables named nunit3-console.exe and nunit3-console.dll, the first being our standard runner and the second being our tool. I don't like this and I'm planning to change it.

In the latest 4.0 dev builds, the only change is to substitute net6.0 for netcoreapp3.1. We still have the same anti-pattern of two applications from one project. I'm focused on other things right now, but I'd like to gather info about exactly what we want a tool to be able to do. Then our net6.0 build of nunit-console can grow to become a full-fledged runner, starting processes, handling extensions, etc. while our dotnet tool can be a different project, possibly with independent code, possibly just wrapping the nunit-console net6.0 build and making use of it in a controlled way.

Everything you see in 3.15, of course, is limited to non-breaking changes I could make.

You're welcome to join in the engine team discussions about where this is going and/or we can set up a phone call some time!

[CharliePoole]

Based on comments... I'm thinking we should treat nunit3-console as an anomaly and just go back to nunit-console (as it was before v3) or to some other name without a version in it.

For now, I think we should avoid using nunit but if it turns out there's no problem using it for two things, we could change again before the final release. So... other ideas for a console/batch runner name for nunit tests?

[siprbaum]

@CharliePoole Doesn't this name change conflict with nunit2, where the runner was also named nunit-console.exe?
I just looked into https://github.com/nunit-legacy/nunitv2/releases/download/2.7.1/NUnit-2.7.1.zip and saw that it indeed uses
nunit-console.exe for its console runner.

[CharliePoole]

Yes, that's what I meant by "go back to nunit-console", treating our use of nunit3-console as an anomaly.

Do you have a specific use case which presents a conflict for you?

[siprbaum]

At least I was under the impression that it is is fine to have e.g. the nunit-console.exe in PATH.
Right now we still have projects on nunit2 and are slowly migrating to nunit3. For that, nunit-console and nunit3-console
worked as their name didn't conflict.

But now reusing the same name (nunit-console.exe) would at least prevent having different nunit-console runner versions (e.g. Nunit2, Nunit3, Nunit4) in PATH and calling it from there.

[CharliePoole]

Fair point. Let me think about this. Meanwhile, if you are testing the 4.0 builds, avoid anything later than 4.0.0-dev00068 unless you run it from a known location.

[CharliePoole]

@jnm2 @OsirisTerje @rprouse @mikkelbu The path issue is kind of obvious I'm afraid. Are we basically forced to use either nunit4-console or some entirely new name?

Alternatively, can we just make sure that the V2 framework driver continues to be supported?

[CharliePoole]

@siprbaum Is there some reason you don't run your V2 tests under nunit3-console?

[siprbaum]

No, not sure, as this is driven by another team.

But we are operating in a regulated environment, so introducing new SW (in that case NUnit.Extension.NUnitV2Driver) is not so easy.

[CharliePoole]

@siprbaum One part of me wants to help you :-) while the other wants companies, which control how their devs develop so tightly, to suffer. :-(

More seriously, if you mean regulated as in medical or other high-risk applications, I've had a great deal of success gaining acceptance for tools, which don't form a part of the product, provided it's clear that they don't generate production code, but are "merely" used to find bugs. Often depends on the individual who is reviewing your work, however.

[siprbaum]

Yes, we fall into the "medical or other high-risk applications" category.

Anyhow, i don't think you should base your decision on our needs here, as we are kind of "special" in that sense.

But I wanted to make sure that at least it was a concious design decision to break the usecase of having different nunit-console versions in your path and not something done by accident.

[CharliePoole]

We'll take it into consideration... right now, I'm waiting to see what other team members think. As you discuss this in-house, you may want to consider that the code for running tests in the V2 Driver Extension is precisely the same code you run under NUnitV2... the legacy V2 project exports an assembly, which is then used by the extension.

[jnm2]

I'm not sure how to choose. Are there other similar command-line tools we could compare?

[stevenaw]

Out of curiosity, what factored into the decision to include the version number in nunit3-console.exe ? Most tools I'm familiar with which have a predominantly CLI interface will often keep the exe name the same across versions (git, dotnet, and 7z for example). I'd imagine one of the benefits of this would be to minimize the need to update any scripts which call it when upgrading major versions. The flipside to this, of course, is that multiple versions of the exe, if desired, would have to live in different directories.

[CharliePoole]

When NUnit V2 was created - before my time - it was made an entirely separate project from version 1, in its own repo. So when I did V3, it seemed logical to do the same thing. Having a separate name for the runner was consistent with that and allowed people to use the old runner over the multi-year development period, which led to the first release of NUnit 3.

The coming 4.0 release of the console runner and engine will actually be first major upgrade within the same project and repository. That's also the case for the coming framework release. Since it's the same repo, having the same name would make sense to me except for the fact that it has "3" in it. :-)

If I call it "nunit4-console" I'm probably committing to having a similar name for 5.0, 6.0, etc. so a name without reference to the version makes sense. In other words, I was hoping to undo what I've come to consider a mistake (using nunit3-console) with nunit4.

OTOH I recognize that this would put an extra burden on any users who need to use multiple versions of "nunit-console", so I'm looking for more input before making the final decision.

[CharliePoole]

I'll summarize the current status of this...

The current dev builds (4.0.0-dev00070 and higher) use "nunit-console" as the name. So folks using 3.x while trying out 4.0 should have no problem but those using 2.x with 4.0 will see conflicts if they rely on the path.

There are some other related issues as well, which I plan to deal with in coming dev builds:

• The msi default install location is the same for 4.0 as for 3.x.
• The default chocolatey install location has the same problem.

For now, my recommendation for those trying out the 4.0 dev builds is to use the nuget.org package to install 4.0 in a tools directory for each project.

[stevenaw]

Thanks for the background @CharliePoole ! That makes sense given the history there.
For what it's worth, as my two cents, I agree with you that continuing to include the version number in the exe would also mean committing to it, but that I also favour going back to a version-agnostic exe name. As noted, side-by-side installs may need to be a consideration there

[manfred-brands]

We haven't used nunit-console since the Nunit2 days. All our tests use dotnet test. As long as the appropriate test adapter for the framework at hand works, different frameworks can co-exist. An other option is to make nunit-console a dotnet tool. Again this allows different repositories use different versions and doesn't rely on having a tool installed somewhere in the path.

[jnm2]

OTOH I recognize that this would put an extra burden on any users who need to use multiple versions of "nunit-console", so I'm looking for more input before making the final decision.

The extra burden is lessened when you consider that there is an option of creating forwarding scripts (like nunit4-console.cmd or symbolic links) and placing those in a folder in the path instead.

[rprouse]

Further to the choice to add the 3 to nunit3-console.exe; the version 3 runner had many breaking command line argument changes AND there were several CI/CD tools that searched for nunit-console and if it was found, passed in the old arguments. We made a decision to change the name to ensure we didn't break those tools and to force them to upgrade.

That said, is this a chance to simplify with nunit.exe?

Like others, I just use dotnet test now though. I only use the console when working on NUnit.

[CharliePoole]

I had forgotten the point about the CI/CD tools automatically using certain options.

Since the change to nunit-console.exe is already merged, I'll leave it for now and let this percolate for a while. We're probably at least two months away from a release.

I'd prefer to reserve nunit for the dotnet tool. With dotnet nunit we could perform similar functions to dotnet test, without requiring the adapter and respecting certain nunit-only conventions like Explicit, so I don't want to rule it out too soon.

Let's come back to this after we decide about the need for the tool.