Post E2E and Apex test results to GitHub Checks

[zivkan]

Bug

Fixes: https://github.com/NuGet/Client.Engineering/issues/2949

Description

• Fix daily pipeline's trigger, and also remove the github checks name, since it doesn't need to report to github checks
• Make apex and e2e templates take an optional github checks name, and when it's non-empty, report the test status
• Make the build template take a list of checks, and for each one, set the github check to pending
• Since NuGet.Client no longer has a single pipeline, there isn't a single list of check names that we'll want to report. It's easier to remove the validation set from the powershell script, than to keep maintaining it.

PR Checklist

• Meaningful title, helpful description and a linked NuGet/Home issue
• Added tests
• Link to an issue or pull request to update docs if this PR changes settings, environment variables, new feature, etc.

[jeffkl]

Somethings wrong the the Apex stage posting its status: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=10317783&view=logs&j=35fd8e9c-c657-57a4-085d-6f9cd010d4b2&t=98636fdc-df00-5b21-8d84-bd3fbb6c1630

I worry that since the end-to-end tests and Apex constantly fail, that it makes our PRs look like they're not ready for merge.

Are these pipelines going to become more stable soon and we don't need to worry about that?

[zivkan]

I worry that since the end-to-end tests and Apex constantly fail, that it makes our PRs look like they're not ready for merge.
Are these pipelines going to become more stable soon and we don't need to worry about that?

That's up to the team. The tests were all passing before we had to switch to 1ES templates a few months ago, we just haven't been monitoring the tests since then. We might have introduced subtle regressions, or maybe it's just test bugs (for example, the underlying VS platform might have changed in ways that our tests can't handle). The team can choose to skip all the failing tests, to quickly get a green pipeline, taking the risk that if we have product bugs, we're not investigating it more quickly.

I don't understand why the VSDaily pipeline reported the overall pipeline status, but the non-daily pipeline did not. If it's configurable, I'd want it to be the other way around, but I would have thought it's not configurable, so either both should report, or neither would.

[nkolev92]

I'm ok with having the checks posted, mostly because it saves me time looking for them through the pipelines :D
IMO without the checks I feel like it might be a bit of a out of sight, out of mind situation. I'm guessing we'll get them green faster if the checks are visible.

[zivkan]

Well, there's only 2 people who commented here, one seemingly to merge as is (to raise awareness of the failing tests), another to skip the failing tests. Since the PR has been open for over a week, I'll cast my vote for raising awareness of failing tests, and merge as is. If the team finds it annoying enough, it's not too hard to skip them in another PR.