Automatic CI for pull requests (progress tracking issue)

[rvagg]

/cc @iojs/build

This is not done yet but I've spent some quality time with my head in code (as a leisure activity!) and have made good enough progress at this point to start turning it on and experimenting with it.

There are 3 components I'm working on:

• GitHub Webhook endpoint that listens to events from the repo and takes action based on the type of event. For pull requests it'll act based on opening a new PR and updating an existing PR. It'll also look up the user that did the PR and decide whether they are "trusted" or "untrusted" based on whether they are in the iojs/Collaborators team on GitHub. It then passes on this data to the jenkins interactor.
• Jenkins Interactor that knows how to speak Jenkins. It can receive authenticated requests for new runs for either trusted or untrusted users on any user/project#branch combo on GitHub. For untrusted users it'll run these in the iojs+any-pr+containers job and for trusted users it'll run them on iojs+any-pr+multi. It also keeps track of the builds on both of these jobs so it knows (a) the status of them including what types of slaves they are run on and the status of each of those slaves and (b) the user/project#branch parameters so you can look up a build based on either a build number or a user/project#branch combo and you'll get the latest build for that combo.
• Status badges for GitHub - this component is the most incomplete but I have a lot of the parts of it in place thanks to nodei.co and all the data is now available in the Jenkins Interactor. The idea is that the bot that interacts with GitHub will either comment on the PR or update the PR head comment to include a badge and that badge would show the current status of the latest build for the user/project#branch combo of that PR including what the status of each of the slaves are. So you'll get to see right in the PR whether you're building, passing or failing on any of the slave.

I've hooked up what I have to this repo and re-delivered a webhook notification for the last PR #1412 and it's triggered a build here: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/497/

The containers build jobs need a lot of love because they just haven't been used and we have problems with git, file descriptors and some other random things that keep them from being useful.

Not sure what the timeline is for getting all of this working but one of my next tasks is to get the code of this up in a repo on iojs so others can contribute. I'll post status updates here but at least for now at least collaborators can trigger CI runs themselves without having full Jenkins access.

[Fishrock123]

The idea is that the bot that interacts with GitHub will either comment on the PR or update the PR head comment to include a badge and that badge would show the current status of the latest build for the user/project#branch combo of that PR including what the status of each of the slaves are. So you'll get to see right in the PR whether you're building, passing or failing on any of the slave.

I'd really rather not have a bot. they are usually very noisy. Can we hook into GitHub's built-in Ci status for PRs?

[rvagg]

yes, that too but it's only a building/pass/fail status indicator and we need the full gamut of slaves to get good feedback -- this is why I'm suggesting editing the original posters' comment and putting the badge at the top, then nobody gets a notification for it.

Something like this at the top of each PR summary:

[Fishrock123]

this is why I'm suggesting editing the original posters' comment and putting the badge at the top, then nobody gets a notification for it.

hmm, that might be do-able.

EDIT: below would be RAD

[jbergstroem]

@rvagg I think the GH api support multiple "checks" per ci run. How about using one for each slave group (ui example here - click green checkmark)?

The downside of top posting a status board is that it'd probably have to contain what revision it last tested. Github sunsets previous test runs, comments, etcetera when new commits are added/force pushed and the "overall" ci/merge status is always at the bottom.

[rvagg]

the top-posted badge only needs to be inserted once and the badge itself can contain details about the run, it's not a static image.

but yes, the multiple status checks is an interesting feature I wasn't aware of! and you even get individual links, let's start with that then because it's probably going to be easier anyway.

[jbergstroem]

@rvagg I was thinking of using it for other steps too; linting, commit message/author checks, etc. The "proxy" would be responsible to delegate different tasks to different workers.

[rvagg]

👍 sounds like a plan

[rvagg]

from the github docs:

Note: there is a limit of 1000 statuses per sha and context within a Repository. Attempts to create more than 1000 statuses will result in a validation error.

so there'll be a hard-limit of 1000 on the number of build slave groups we can have for io.js, good to know.

[Fishrock123]

Note: there is a limit of 1000 statuses per sha and context within a Repository. Attempts to create more than 1000 statuses will result in a validation error.

That almost sounds like the combination of how many checks we make and how many times we make them on a PR must be less than 1000..

(Not that that will be an issue really)

[rmg]

The "context" is the label used for a commit status, and updating a status with a context that has already been set overwrites the previous value.. I'm pretty sure if a PR had that many statuses set on its head commit there would be complaints from devs looking at it long before we hit 1000. Cool to know the limit is high enough to allow for other kids of automation/workflow, though.

[ChALkeR]

@rvagg Could it be made partial? Lint run is fast, for example. Even having just lint checks as status badges would be good.

[Trott]

Is this still something that's wanted? If so, is there anything actionable at this time? Maybe some progress has happened that wasn't noted here?

[jbergstroem]

@Trott a lot of partial progress has been made, but for now the conclusion is pretty much that we shouldn't proceed with automatic testing. I have other ideas/plans for linting and perhaps even commit message validation though!

[jbergstroem]

A few things is starting to brew over here: https://github.com/nodejs-github-bot/github-bot/issues

[Trott]

@jbergstroem Any objection to closing this issue?