Consider using NuGet release of Console to run tests in CI

[ChrisMaddock]

#779 is a potentially major issue, and meant our own tests were not showing failures for several months. Fortunately there were no problems this time, but it does highlight a potential flaw in using the under-development test runner in our own CI.

Would it be more robust to use the latest NuGet release of the Console runner, to run our own tests? I can't think of any reason that wouldn't work - anyone else?

[CharliePoole]

Seems like a valid approach so long as the package is isolated... i.e. no components get copied to the output directory.

Would it be worthwhile to have a non-default --self-test switch on the build to be used to run the actually built runner for final testing?

[ChrisMaddock]

Would it be worthwhile to have a non-default --self-test switch on the build to be used to run the actually built runner for final testing?

Could you elaborate on what you mean here? 🙂

[CharliePoole]

I'm imagining using the NuGet package to run the tests for

./build.sh -t Test

while this command

./build.sh -t Test -- --self-test

would revert to the current behavior of using the console build we just created.

Alternatively, it could be a separate target, like

./build.sh -t SelfTest

[ChrisMaddock]

Ahh, I see what you mean. A kind of, final-smoke-test thing?

[CharliePoole]

I'm inclined to close this without action. While it's possible to use the stable version of the console runner to test the new version we are building, it introduces complexity, which we don't particularly need.

If we want to be completely sure that the engine tests are not influenced by the console runner in use, I'd favor switching all of them to NUnitLite. What do others think? Shall I close this? Do we need another issue for using nunitlite?

[mikkelbu]

I agree that we should close this one - and I don't mind switching the other two engine tests to NUnitLite.

[CharliePoole]

Replaced by #1087