Develop a new CI build running under GitHub Actions

[CharliePoole]

AppVeyor is continuing to have issues so creating an alternative approach is necessary. This should be relatively simple since all it has to do is kick off our build.cake with the proper options and handle the results.

@veleek Are you familiar with GitHub actions? Would you be willing to take this?

[CharliePoole]

It appears that @OsirisTerje has started this already. We can use this as a basis. I am inclined to leave the logic of what actions to take to the Cake recipe rather than calling cake multiple times for a push, but that's open for discussion.

[CharliePoole]

@veleek Any progress on this?

[veleek]

Minimal progress, mostly spending time digging through the existing code and familiarizing myself with build stuff.

I spent a bit of time getting stuff setup as well as trying to see if I could use act to run the actions locally, but that ended up being a dead end because it doesn't work great with Windows hosts (what the current action uses).

[CharliePoole]

I moved it the next release in order to get 3.18 out.

Act sounds interesting. Too bad it didn't work out. So what next? Are you able to continue on this?

I'm wondering if it isn't as simple (initially) as looking at what we do in our appveyor.yml and emulating it. The cake script is basically a black box to AppVeyor except for the fact that it has an AppVeyor target, which does what we want done on AppVeyor. I can create an initial target GitHubActions, and set it up initially to do just Build and Test.

Going beyond those two steps introduces questions of authorization of course, but there is a --nopush option in the script to use in testing without actually publishing new releases. Cake itself has a --dry-run option (possibly mis-spelled) we could also use for tests - it merely lists what would have been done.

[veleek]

That works for me! I had a bit of time to spend on this tonight and I'm testing a bunch of stuff out in my fork, but I'm running into some unexpected issues.

I have the action setup to just run the exact same build.ps1 that is run by the Appveyor script and it's failing on a later packaging step. See https://github.com/veleek/nunit-console/actions/runs/10002323808/job/27647413350 for details on the error.

Attempting to gather dependency information for package 'NUnit.Extension.NUnitProjectLoader.3.7.1' with respect to project 'D:\a\nunit-console\nunit-console\package\tests', targeting 'Any,Version=v0.0'
An error occurred while retrieving package metadata for 'NUnit.Extension.NUnitProjectLoader.3.7.1' from source 'https://community.chocolatey.org/api/v2/'.
  Failed to fetch results from V2 feed at 'https://community.chocolatey.org/api/v2/Packages(Id='NUnit.Extension.NUnitProjectLoader',Version='3.7.1')' with following message : Response status code does not indicate success: 504 (Gateway Time-out).
  Response status code does not indicate success: 504 (Gateway Time-out).

I'm surprised by a Gateway Time-out, but I also can't find that package at all, since https://community.chocolatey.org/packages/nunit-extension-nunit-project-loader/3.7.1 definitely exists. Do you have any insight about what to do here?

Mostly unrelated thing, I also wanted to see whether we could use the GitHub actions "matrix" strategy to build/test a bunch of the different versions in parallel. It'd probably need to happen later because to do it right we'd have to pull a bunch of the logic out of cake and into the GitHub action, but before I spend time on it i'm interested in hearing whether there's a reason that we need to build them all together or could we pass in a "framework version" attribute and have it build/test that one. Or are there tests that depend on having multiple different versions SxS?

[CharliePoole]

That package is not a chocolatey package... I think nuget is just looking on all the feeds for it and happens to encounter that gateway timeout.

I've never found a problem with building sequentially. I'm in the "make it work first" camp.

[CharliePoole]

@veleek As I mentioned, I'll be (mostly) out of action for the next three weeks. My sister and her husband are visiting us.

I'll do a bit, perhaps late at night, with my priorities being...

1. Any new critical bugs that pop up for the 3.0.1 NUnit-Console release
2. Some experimental scripting changes on TestCentric.

My suggestion for some sub-goals for you on this issue:

1. Get an action working that specifies target "Test" (which implies Build as well). Based on your latest output, that should work easily, because Package would not have executed if Test had not succeeded.

2. Try a new one, which t targets either "BuildTestAndPackage" or just "Package" (implying Build).

3. Move on to "Publish", which implies BuildAndPackge. For safety, you should add "--nopush" to the arguments for build.ps1 so that no actual publishing occurs. You can verify that it ran correctly by looking at the output for a message that tells you pushing was suppressed.

You should rebase your own branch on the latest main, which matches the 3.0.1 release. Since I wont be changing the script for a few weeks it will remain stable. Of course, if something critical requires changing the script itself, I'll tell you,
but that's unlikely.

Time allowing, I'm going to port some script changes I made on NUnit to TestCentric. In particular, I added code to allow package tests to be run against multiple versions of the console, which is useful for testing extensions. I may also shadow some of your work on TestCentric.

[veleek]

Created an initial PR to make some tweaks to the github action, as well as a separate one to update the Cake recipe to enable binlogs.

Next step is going to be further troubleshooting the package steps to try and identify the problems.