Migrate Jenkins job configuration to pipeline files

[maclover7]

Currently, it looks like all Jenkins jobs are configured through the web interface. It would be advantageous if all of the jobs were managed through pipelines (a Jenkins 2 feature), which allows for job configuration to be checked into version control. This would allow for more transparency in the configuration, and using a normal code review process to approve changes.

Some key points to consider are...

• where should the pipeline files live? should they be in the project itself (travis-style), or should they live in nodejs/build in a jenkins directory?
• migrating the jobs with no downtime
• since job configuration is being checked into source control, is there a better way to manage admin groups/users, with the additional metadata?

If this all sounds alright, I can try and draft up a more complete project plan and migration strategy :)

[gibfahn]

I've been thinking that this would be a good idea for a long time (and it just came up in #804 (comment)). It's a fairly massive change though, as pipelines are pretty different from the existing build trigger plugins and methods.

where should the pipeline files live? should they be in the project itself (travis-style), or should they live in nodejs/build in a jenkins directory?

I think I agree with @joaocgreis 's point in #804 (comment), we want to have one master copy of the build scripts, not one for each version, so it makes more sense to have them in a separate repository. I think a jenkins/ directory in this repo might make sense.

since job configuration is being checked into source control, is there a better way to manage admin groups/users, with the additional metadata?

I think the current system of using Github teams to manage access works quite well, but if Jenkins has some alternative method of managing this we should definitely consider it.

migrating the jobs with no downtime

I think the first thing we need to look at is whether this is worth doing, and how much work it's going to entail. Node CI includes non-core CI tests (like CitGM, node-report, and libuv). Those would be a good place to start, if we convert some of those jobs, then that would give us a good indication of how difficult/worthwhile it's going to be.

Another issue is that no-one in the build WG is a Pipeline expert. Many of us are familiar with the existing (Jenkins 1) configuration methods, but pipelines are a new way of doing things.

[maclover7]

I think a jenkins/ directory in this repo might make sense.

I agree -- I think having them in nodejs/build alongside other relevant configuration (ansible, etc) would be helpful

Node CI includes non-core CI tests (like CitGM, node-report, and libuv). Those would be a good place to start, if we convert some of those jobs, then that would give us a good indication of how difficult/worthwhile it's going to be.

Would definitely be worthwhile to do some beta testing with some of the smaller jobs before trying to take on larger ones. I created a small CLI utility to pull some metadata about the cluster, and it looks like there are a bunch of freestyle jobs that might be good candidates to experiment with. Without knowing too much or having access to the CI admin portal, it looks like the benchmark tasks might be on the easier side to port.

Another issue is that no-one in the build WG is a Pipeline expert. Many of us are familiar with the existing (Jenkins 1) configuration methods, but pipelines are a new way of doing things.

I've done some work with pipelines/Jenkinsfiles in the past, but definitely not going to say I'm an expert 😅

[joaocgreis]

This would be great, if pipelines can replace all other types of jobs.

Having no downtime would be nice, but we could do with some downtime during the weekend or even during the week at nighttime.

Also note that much of what we do is reacting to issues, and we need to be able to change the jobs quickly without going through a review process. It would be good to have a review process after the fact: it's good for notifying other WG members of what changed and gathering suggestions for improvement. But we need a way of reacting fast to issues.

[gibfahn]

and we need to be able to change the jobs quickly without going through a review process

@joaocgreis agreed, I assume we'd have a rule that fixes for broken jobs can be landed without review (although if someone is around to take a quick look that's always good). As you say, having the PR is helpful even if it lands straightaway.

I'll put this on the agenda as well, we're meeting in a couple of days, and it'd be good to get everyone's thoughts on this.

[rvagg]

I'm torn on this. On one hand I'm strongly in favour of getting as much config out of Jenkins as possible and the new github-hosted pipelines are really nice for this (and Groovy can be quite expressive if done well .. although it can be equally terrible). On the other hand, I've not had fantastic experience with the new Jenkins pipeline build processes, it feels like poorly implemented hack on a framework that was never designed to work that way. It could very well be that I've just not seen a well implemented Jenkins pipeline build project that's elegant and demonstrates the full potential of the feature. Have any of you? Perhaps this the way forward here is to simply implement some minor part of our CI with it and see how it goes side-by-side with our existing setup.

[mhdawson]

I'm in favor of trying out, possibly on the jobs for one of the Work Groups.

[refack]

I'll try to do #844 as a POC

[gibfahn]

@maclover7 FYI, we discussed this in the meeting (minutes in #845, recording here). Agreement was basically what was already said, people are positive in theory, but unsure how well things will work in practice. So we should try to do it gradually, one job at a time would be good.

It's a bit difficult to work out how you could contribute (that's one of the things that will be much easier if we do this).

@refack maybe you could raise a PR adding a pipeline to /jenkins/ in this repo, and run the CI job from that PR, that way people could review and suggest (and see it working). Ideally as @mhdawson said, we'd have a generic pipeline that did "pull down node, run a custom extra setup script if necessary, git clone project, npm install, npm test", that could work for other CI jobs as well.

[maclover7]

@gibfahn watched the meeting, good discussion about pros/cons. Just wanted to clarify that pipelines are implemented as a Jenkins plugin, so should be unrelated to Java version on build machines. Also, agree about strategy as discussed earlier about converting over jobs slowly, to make sure the end result is helpful.

we'd have a generic pipeline that did "pull down node, run a custom extra setup script if necessary, git clone project, npm install, npm test", that could work for other CI jobs as well.

While I think this would be great, most CI jobs don't run in this cookie-cutter fashion, and I'm not sure if this is practical. I think it makes sense to convert jobs over by "batch" (good example of this are the benchmark jobs)

I'm not sure how Jenkins permissions are assigned to contributors, but I'd be willing to take a look at pipeline-izing the various benchmark jobs, since those seem fairly straightforward.

[rvagg]

@maclover7 good to hear re plugin, that makes this safer to tackle I think. Currently we only assign admin permissions to a smallish group under the @nodejs/jenkins-admins team and we probably shouldn't go much wider because there are broad security implications (simply in access and ability to mess around but also in providing too much access to details of security patches that are under test). However, we should be able to do it per-job so my guess is that we could make a job (or two, or three) and assign you and @refack as admins of it so you can tinker with it all you need to get it to a right state. I don't know if our per-job permissions matrix will show up for pipeline jobs but we could give it a try. @refack: want to propose a job name for #844 and we can try and set that up in Jenkins as a Pipeline, pointing to a new test repo for now perhaps?

[joaocgreis]

I learned a bit about pipelines and started experimenting, but found one big issue. There is no good way to visualize jobs that use a matrix (like current multi-configuration jobs, such as node-test-binary-windows). Currently, we have a table of configurations displaying green/red icons, and the console outputs can be displayed individually. However, with a pipeline job, everything would be mixed in the console and there would be no table in the job run page. This is an issue tracked by Jenkins: https://issues.jenkins-ci.org/browse/JENKINS-33185

Current pipeline jobs (like citgm-smoker-pipeline) already have this issue. Clicking on any run will not show the jobs started by it, we need to see the console output to find the links. Since this only starts other jobs, there is no output mixed in the console, but this would be a problem otherwise.

If not for this issue, I believe pipelines would be a major improvement for us. If that issue is solved in Jenkins or if there's some solution that I did not find, pipelines would certainly improve our workflow.

[gibfahn]

@joaocgreis that page says that Blue Ocean (apparently a UI rewrite and a Jenkins plugin) has some nicer UI for looking at multi-config jobs. It might be useful to look at that.

[mklebrasseur]

Hey everyone, I'd love to assist here if at all possible.

We recently converted all legacy builds and DSL scripts to pipelines using shared libraries (also using blueocean). We use source control and branching to control our releases from shared libraries. We ran these in parallel with (no downtime, no risk config).

We used this model to control granular access to certain modules to where access was needed. All secrets are encrypted and refereced as environment variables. Allowing us to use a single pipeline for multiple projects. As well as allowing for a more standard infrastructure as code feeling.

I have only needed the UI to check failed logs, we use another plugin GitHub source branch that will scan an entire organization for Jenkinsfiles so that you never need to add another job to Jenkins.

Can go in to greater detail but it cut a lot of our repetitive code more than 50%.

We now only use pipelines for environments. It takes far less effort to maintain so many pipelines and resources etc.

[sam-github]

Closing as stale, but if anyone wants to take this up feel free to reopen.