CI status updates to GitHub stuck at yellow

[refack]

nodejs/node#14067

https://ci.nodejs.org/job/node-test-commit/11055/

[phillipj]

Dumping out some observations while digging into this:

• Last bot deploy 3rd July
• All the latest statuses updates from Jenkins sets status: pending, resulting in the yellow status in GitHub PRs
• Has to go two days back, to 10th July, to find Jenkins updates which set a status else than pending, meaning failure|success.

As the bot hasn't gotten any recent changes within the last two days, I'm suspecting something Jenkins related at the moment.

[phillipj]

Seems like the post build step configured to report the build result to the bot via curl doesn't exist anymore. The same is done to report the pending status when a build starts, easily spotted when looking at a build configuration, an "Execute shell" block sending a JSON payload via curl with a static status value of pending.

Pretty certain the same thing was done to report failure|success, configured as two post build steps per build. @jbergstroem can confirm that sounds familiar?

@nodejs/jenkins-admins has something particular been done to Jenkins or the nodejs/node related builds the last couple of days?

[joaocgreis]

@phillipj I can confirm that is correct. The build step to mark as failure|success disappeared. This is most likely because the Jenkins plugins were updated this week (while Jenkins was locked down). It's far from the first time we've had issues after updating Jenkins/plugins.

I took a look at the OSX job (picked randomly - any would have worked). The last modification was in 2016. The script to mark as failure|success in not there, there is only an empty Flexible Publish step. So I saved, without modifying anything, and the log confirms that Jenkins removed those steps on its own.

Thus, something in some plugin changed and what we were doing is no longer supported. We must find a new way or better understand what happened.

[Fishrock123]

ping @nodejs/build

[phillipj]

Looks like we used the PostBuildScript plugin which was suspended in June and I cannot seem to find any replacement. I have no idea how to execute shell commands post build without that plugin in place.

[joaocgreis]

I can't explore this now, but someone else might be able to pick this up. If we create a new job with the script to post a success/failure to GH (based on a parameter), we may be able to use Trigger parameterized build on other projects to start it and post the status.

[maclover7]

ping @nodejs/build... is there anyway I can help with getting this working again?

[refack]

@maclover7 if you have an idea how to add a post-build-command to all the jobs, or if you can investigate the status of PostBuildScript plugin (maybe it's vulnerability does not affect our setup).

[phillipj]

The third alternative, if we can't get Jenkins to do as before, is to implement build polling in the bot. That's a lot more hassle and complexity in the bot, but an alternative nonetheless.

…

On Sat, 14 Oct 2017 at 17:20, Refael Ackermann ***@***.***> wrote: @maclover7 <https://github.com/maclover7> if you have an idea how to add a post-build-command to all the jobs, or if you can investigate the status of PostBuildScript plugin <https://wiki.jenkins.io/display/JENKINS/PostBuildScript+Plugin> (maybe it's vulnerability does not affect our setup). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#790 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABLLE_xE3o_uawdDMWsq-WsR8XKfW_09ks5ssNE3gaJpZM4OU_LA> .

[maclover7]

The Post build task plugin might also be an alternative. Maybe we can try out one solution, see what happens, and then go from there? Unfortunately I'm not a member of the Build WG, but maybe a sandbox job could be created to test this out.

[gibfahn]

The Post build task plugin might also be an alternative. Maybe we can try out one solution, see what happens, and then go from there? Unfortunately I'm not a member of the Build WG, but maybe a sandbox job could be created to test this out.

@maclover7 this really needs someone to test it with their own repo and get something that works. Once we have that we can just PR changes to the github-bot repo (and someone can change whatever secret is needed in the Jenkins config).

[maclover7]

#973 has been rolled out to all node-test-commit sub jobs, so hopefully this should be resolved!

@nodejs/collaborators green/red CI status lights should be appearing now in PRs... if you encounter any errors please post here on this issue to try and keep the discussion all in one place. Unless any huge errors come up, will close out this issue in a few days ❤️

[maclover7]

This largely seems fixed and to be working, going to close for now :)