build: use Actions to validate commit message

[mmarchini]

Actions interface has a better integration with GitHub, and with
Annotations and Problem Matcher we can display all failed checks in a
single place, so that users don't have to go through the logs to figure
out what's wrong. Since the job on Travis was allowed to fail and is not
as easy to read, remove it from our Matrix.

The Action will check every commit in the Pull Request, skipping commits
with "fixup" or "squash".

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[mmarchini]

Annotations in action:

I tried to treat the commit as filename by creating files with the commit messages in the Action, but GitHub does not allow files that are not committed in the Pull Request.

[addaleax]

I’m not entirely clear on whether it works this way or whether the JSON values all have to refer to regexp capture groups, but could we use something like "severity": "warning" (or "severity": "notice"? not sure if that would work either) so that this does not appear like a failed build, given that this is ultimately not essential for PR authors to address?

[mmarchini]

The severity only works for annotations, the Check status can only be Success or Failure

[addaleax]

That’s unfortunate :/ Any ideas for how else we could make clear that this is non-essential? I’d rather not have explicit failures because of this…

[mmarchini]

We could prefix the workflow name with "[not a blocker]" or hijack the annotation to have this as a message. Unfortunately I don't think there's anything we can do using GitHub's features (the best thing would be for the Action leave a comment, but we can't do that with Actions for PRs coming from forks), and I would rather not have the check on CI than mark as "allowed to fail", because if we do so it'll be misleading for newcomers (the check will always be green).

[mmarchini]

Switching to warning instead of error on the annotation is probably still worth though. I'll update the PR

[mmarchini]

Hum, reading the Checks API docs again it seems there a "neutral" status.

https://developer.github.com/v3/checks/runs/

I never seen it, and I'm not sure we can set it without directly calling the API (which is not possible in the Action), but it's worth a shot.

(There's also a "Action Required" status, but it's visually worse than "Failure" IMO)

[mmarchini]

GitHub Actions has no way of setting the "neutral" status on runs without calling the Checks API directly (which needs a write-permission token, which means it wouldn't work on Pull Requests from non-collaborators). So, we can either:

1. Have the check fail
2. Set it to continue-on-error
3. Don't check the commit message on CI

I'd rather not set it to continue-on-error since any failures will fly under everyone's radar. I'm fine not landing this change though, if folks think failing the CI because the commit messages don't follow our guidelines is unwelcoming. Personally I think the annotations are helpful for folks who want to make sure their commits are following the guidelines. Maybe we could force one annotation with a "not required" message, but I'm not sure how useful and clear that would be.

On the other hand we could also use this check to verify Signed-off-by metadata on commits once #31976 lands, or have another Action doing the same verification. Either way the CI would fail if the commit is not signed, so maybe it's a good thing to introduce this checks now instead of later.

One alternative I've seen in the past is keeping the check "pending" when things like Signed-off-by are missing, but then again, I don't think this is possible with the current Actions permission model.

[targos]

How can we move forward with this?

[mmarchini]

If we want to move forward as-is (fail the build), I just need to rebase. Otherwise I think we should drop it, since GitHub Actions doesn't allow us to set an intermediate state, and using continue-on-error would give a false sense of the commit message being formatted correctly (which IMO is worse than not having the check).

[mmarchini]

We should be able to write a better Action once nodejs/github-bot#272 lands, as it will allow us to set the Check state to neutral or action_required (might need to create a new Check entry, I'm not sure) as well as leave a comment with suggestions to fix the commit message. Blocking until then.

[mmarchini]

I just tested if it was possible to use pull_request_target for this, and it is! We can create an extra Check using the Checks API, and it will propagate the status to the "parent Action":

We can also add a summary of guideline violations for when the user clicks the "details" link in the Action. This is probably the best we can do with Actions to lint the commit message without a big red X in the PR.

These are the options we have for status:

In the example above I used Neutral. We could use Action required (not a big red X but still a big red icon), or we could leave the check pending/in progress (it should show as a yellow dot). We can add the summary either way so that's not a big deal, we just need to choose which status to use.

cc @addaleax since you were interested in this before :)

[mmarchini]

I'll go ahead and implement the above with "Action Required". With the commit queue I believe it's more relevant to show when the commit message is not following our guidelines (and therefore will prevent us from using the commit queue). This signal the contributor that their PR might take more time to land compared to a PR following the commit guidelines.

If Action Requested turns out to be too disruptive or drives contributors away, we can revisit and move to Neutral or Pending.

[mmarchini]

Revisiting this PR and I'm inclined to move forward with the current implementation. Using neutral status would be the same as not having the linter, and setting action required would require a custom script that significantly increases maintenance burden of the linter. I believe this validation is important if we want commit messages to follow our guidelines because otherwise we block collaborators from using the commit queue on certain PRs and force them to fallback to using ncu in the command line to land pull requests when those pull requests have commit message linter failures.

cc @nodejs/tsc since this affects the collaborators (especially new collaborators) experience.

[aduh95]

Suggested change

	- name: Use Node.js 12
	uses: actions/setup-node@v1
	with:
	node-version: 12.x
	- name: Use Node.js ${{ env.NODE_VERSION }}
	uses: actions/setup-node@v2
	with:
	node-version: ${{ env.NODE_VERSION }}

[aduh95]

I'm unsure if this syntax is valid YAML, but you can (and should IMO) use the actual number of commits.

Suggested change

	# Last 100 commits should be enough for a PR
	fetch-depth: 100
	fetch-depth: ${{ github.event.pull_request.commits }} + 1

[aduh95]

Suggested change

	env:
	NODE_VERSION: 14.x

[addaleax]

👍

[Trott]

🚀

[mhdawson]

LGTM

[mmarchini]

Landed in 53a63de

[richardlau]

This failed? https://github.com/nodejs/node/pull/39120/checks?check_run_id=2887829048 and https://github.com/nodejs/node/runs/2887719401?check_suite_focus=true.

[richardlau]

Looks like a rebase is needed for the PR's those actions ran on.

[targos]

Shouldn't we actually use the pull_request_target event?

[mmarchini]

@targos shouldn't be needed since we don't use anything that needs extra privileges.

@richardlau hmmm, so it seems like GitHub Actions doesn't rebase/merge PRs on pull_request event? That's concerning since it would also affect our tests. I wonder if checkout has an option to rebase when/after cloning.

[targos]

shouldn't be needed since we don't use anything that needs extra privileges.

Not for the privileges, but to ensure the matcher is available.

[mmarchini]

Found the issue, I set ref on checkout here which causes the checkout step to fetch only PR HEAD and not the merged/rebased commits. I think it'll still work if we just remove ref, but I'm not sure. Worth trying. I'll open a PR.

[mmarchini]

#39121