Tracking Jenkins changes #2133
Comments
|
I believe the Jenkins approach to tracking changes is to use pipelines and jenkinsfiles (and to store/checkout at build time the jenkinsfile under source control). There are challenges involved with using pipelines (the web UI is quite different, for example). The alternative are plugins like the https://plugins.jenkins.io/jobConfigHistory which we have installed but doesn't appear to be very reliable. We kind of went with the worst of both worlds in our previous attempts to use pipelines (i.e. we had the different UI but weren't benefiting from tracking as we didn't use jenkinsfiles under source control (e.g. https://github.com/nodejs/build/blob/master/jenkins/pipelines/node-test-pull-request-lite-pipeline.jenkinsfile is not a live version of the pipeline script embedded in the https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/ job)).
We do have the One of things I've noticed is that sometimes the issues are reported in repositories other than build (e.g. reported in core nodejs/node or CITGM issues sometimes reported in nodejs/citgm). Or sometimes a single issues spawns several config changes. |
|
pipelines sounds really promising. Like many things that are are substantial promising improvements, its not a quick fix, it takes someone volunteering to do the work. |
|
Back when @gibfahn was still working at IBM we looked into pipelines for a bit but it did seem that to get full value out of it you had to fully commit (as in write the whole end-to-end sequence as a pipeline) rather than just use a pipeline to orchestrate other non-pipeline jobs. I've also not seen a good visual UI for Jenkins for visualizing parallel multiplatform builds (pipelines or not). |
|
I've been thinking about an automated way of tracking changes that sits outside of the Jenkins UI for some time. My idea is something like this: run a cron job that checks for changes in config.xml on the Jenkins servers. Changes get committed into a git repository that contains just the configs for all the jobs. The git repo gets pushed to a private remote in this org, the repos are accessible to the team that has admin access to the instance, jenkins-admins and jenkins-release-admins. There's an element in the XML that has some provenance info, a |
|
Not a bad idea... I recall there was a plugin for syncing the xml to a source code repository and went looking for it but it turns out to be deprecated (for pipelines) and has not been updated for three years: https://plugins.jenkins.io/scm-sync-configuration Internally in IBM when we were building our own Node.js binaries we were using https://plugins.jenkins.io/thinBackup in our Jenkins set up and then committing the backups to a git repository. (Again this appears to have not been updated for three years.) I was browsing the issue tracker for the job configuration plugin to see if there was anything reported/solutions that might explain it's weirdness in our installation and there's https://issues.jenkins-ci.org/browse/JENKINS-48677 which suggests the plugin gets confused by changes outside of the UI (e.g. made directly to the config xml files). |
|
we don't make changes directly to the config files, but we do delete builds on the disk and I believe our "force reload" is broken because of the crumb thing, so perhaps it's to do with that |
|
Here's my first pass at a shell script, with git remotes commented out for now. This script is located inside a git repo at the root. I'll stick it in a 5 minute cron or systemd timer (I think these machines are too old for systemd timers tho). It copies jenkins config.xml files into the repo under a subdirectory "jobs", then if there are changes it commits them. When we have a repo to work with this (well, two repos, one for each server), then it'll also first do a sync with the remote (so we could update the script and make other changes on the remote) and then do a push if there are changes later. When we have the repos we'll also need to figure out how to give this thing access. Maybe deploy keys are good enough for this or do we need to log in with a GitHub account, I think we have at least one for the org. #!/bin/bash
set -e
__dirname="$(CDPATH= cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ensure we are in sync with remote
git clean -fdx
git reset HEAD --hard
git checkout master --force
# git pull -r origin master
# the ** replacements are to deal with spaces in names which `for` can't handle but jenkins jobs can
for job in $(find /var/lib/jenkins/jobs/ -maxdepth 2 -name config.xml | sed 's/ /**/g' | awk -F/ '{print $6}'); do
job=$(echo "$job" | sed 's/\*\*/ /g')
src="/var/lib/jenkins/jobs/${job}/config.xml"
dst="${__dirname}/jobs/${job}.xml"
cp "$src" "$dst"
done
if [ -n "$(git status --porcelain $__dirname)" ]; then
# there are changes, push them
git commit $__dirname -m "ci: auto update"
# git push origin master
fi |
|
Request to create the 2 new repos: nodejs/admin#464 |
|
In the last Windows updates batch, I added a half-step towards tracking. Jobs are still normal jobs, but everything in the Build section is committed to https://github.com/nodejs/build/tree/master/jenkins/scripts. The Build section becomes simply This makes it simple to switch to a fork for testing, while hopefully giving some motivation to PR more permanent changes properly in the build repo. Another big advantage is reusing scripts across jobs. The downside is that there is still much left in the job. The proposal here looks good as a step to cover the rest of the job. We won't get commit messages explaining the changes, so having both mechanisms seems like the best option for now. We have several categories of jobs in Jenkins, and I believe we should treat them differently. For all the jobs under |
|
Confirmed the backup script has been collecting changes in the last week since I turned it on, working well: Commit from earlier today$ git show 116bb180fad3a937fa0ef9d2c4dc9480289e8c53
commit 116bb180fad3a937fa0ef9d2c4dc9480289e8c53
Author: Node.js Jenkins (ci.nodejs.org) <build@iojs.org>
Date: Wed Jan 29 10:30:01 2020 -0500
ci: auto update
diff --git a/jobs/node-test-node-addon-api-new.xml b/jobs/node-test-node-addon-api-new.xml
index cb0c459..e2a824b 100644
--- a/jobs/node-test-node-addon-api-new.xml
+++ b/jobs/node-test-node-addon-api-new.xml
@@ -135,7 +135,7 @@ To test a pull request use: refs/pull/999/head.
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
<value>
- <time>1579130180141</time>
+ <time>1580311770556</time>
<timezone>America/New_York</timezone>
</value>
<checked>false</checked>
@@ -152,7 +152,7 @@ To test a pull request use: refs/pull/999/head.
<parent class="metadata-tree" reference="../../.."/>
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
- <value>Richard Lau</value>
+ <value>Ash Cripps</value>
</metadata-string>
<metadata-string>
<name>full-name</name>
@@ -160,7 +160,7 @@ To test a pull request use: refs/pull/999/head.
<parent class="metadata-tree" reference="../../.."/>
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
- <value>Richard Lau</value>
+ <value>Ash Cripps</value>
</metadata-string>
</children>
</metadata-tree>
@@ -255,7 +255,6 @@ case $nodes in
ppcle-ubuntu*) OS=linux; ARCH=ppc64le; EXT=tar.gz;;
*-s390x) OS=linux; ARCH=s390x; EXT=tar.gz;;
aix*) OS=aix; ARCH=ppc64; EXT=tar.gz;;
- ppcbe-*) OS=linux; ARCH=ppc64; EXT=tar.gz;;
win*) OS=win; ARCH=x64; EXT=zip;;
smart*) OS=sunos; ARCH=x64; EXT=tar.gz;;
esac
@@ -278,7 +277,6 @@ case $nodes in
ppcle-ubuntu*) OS=linux; ARCH=ppc64le; EXT=tar.gz;;
*-s390x) OS=linux; ARCH=s390x; EXT=tar.gz;;
aix*) OS=aix; ARCH=ppc64; EXT=tar.gz;;
- ppcbe-*) OS=linux; ARCH=ppc64; EXT=tar.gz;;
win*) OS=win; ARCH=x64; EXT=zip;;
smart*) OS=sunos; ARCH=x64; EXT=tar.gz;;
esac
diff --git a/jobs/node-test-node-addon-api.xml b/jobs/node-test-node-addon-api.xml
index 33ee2cc..77e12d0 100644
--- a/jobs/node-test-node-addon-api.xml
+++ b/jobs/node-test-node-addon-api.xml
@@ -4,7 +4,7 @@
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
- <hudson.plugins.jira.JiraProjectProperty plugin="jira@3.0.10"/>
+ <hudson.plugins.jira.JiraProjectProperty plugin="jira@3.0.11"/>
<hudson.security.AuthorizationMatrixProperty>
<inheritanceStrategy class="org.jenkinsci.plugins.matrixauth.inheritance.InheritParentStrategy"/>
<permission>hudson.model.Item.Build:nodejs*n-api</permission>
@@ -133,7 +133,7 @@ To test a pull request use: refs/pull/999/head.
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
<value>
- <time>1572151071922</time>
+ <time>1580311721251</time>
<timezone>America/New_York</timezone>
</value>
<checked>false</checked>
@@ -150,7 +150,7 @@ To test a pull request use: refs/pull/999/head.
<parent class="metadata-tree" reference="../../.."/>
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
- <value>Rod Vagg</value>
+ <value>Ash Cripps</value>
</metadata-string>
<metadata-string>
<name>full-name</name>
@@ -158,7 +158,7 @@ To test a pull request use: refs/pull/999/head.
<parent class="metadata-tree" reference="../../.."/>
<generated>true</generated>
<exposedToEnvironment>false</exposedToEnvironment>
- <value>Rod Vagg</value>
+ <value>Ash Cripps</value>
</metadata-string>
</children>
</metadata-tree>
@@ -189,8 +189,7 @@ To test a pull request use: refs/pull/999/head.
<string>ubuntu1404-64</string>
<string>ubuntu1404-32</string>
<string>ubuntu1604-64</string>
- <string>rhel72-s390x</string>
- <string>ppcbe-ubuntu1404</string>
+ <string>rhel7-s390x</string>
<string>smartos18-64</string>
<string>smartos17-64</string>
<string>smartos16-64</string>
@@ -241,7 +240,6 @@ case $MACHINE in
ppcle-ubuntu*) OS=linux; ARCH=ppc64le; EXT=tar.gz;;
*-s390x) OS=linux; ARCH=s390x; EXT=tar.gz;;
aix*) OS=aix; ARCH=ppc64; EXT=tar.gz;;
- ppcbe-*) OS=linux; ARCH=ppc64; EXT=tar.gz;;
win*) OS=win; ARCH=x64; EXT=zip;;
smart*) OS=sunos; ARCH=x64; EXT=tar.gz;;
esac
@@ -284,7 +282,6 @@ case $MACHINE in
ppcle-ubuntu*) OS=linux; ARCH=ppc64le; EXT=tar.gz;;
*-s390x) OS=linux; ARCH=s390x; EXT=tar.gz;;
aix*) OS=aix; ARCH=ppc64; EXT=tar.gz;;
- ppcbe-*) OS=linux; ARCH=ppc64; EXT=tar.gz;;
win*) OS=win; ARCH=x64; EXT=zip;;
smart*) OS=sunos; ARCH=x64; EXT=tar.gz;;
esac
@@ -388,7 +385,7 @@ fi
</org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder>
</builders>
<publishers>
- <hudson.tasks.Mailer plugin="mailer@1.29">
+ <hudson.tasks.Mailer plugin="mailer@1.30">
<recipients>michael_dawson@ca.ibm.com gabriel.schulhof@intel.com Arunesh.Chandra@microsoft.com</recipients>
<dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>
<sendToIndividuals>false</sendToIndividuals>I've also set it up on ci-release now so we are collecting that too. Can add a |
|
OK, both repos are now being updated as part of the backup script. Each run pulls the latest from the repo, runs a copy of config.xml files, then if there's changes makes a commit and pushes to the repo. The backup.sh script is in the repo so it can be updated from there, but that makes it a security concern so we need to guard changes to it fairly carefully (or maybe reconsider this mechanism). Next step is to do an audit of the kind of data contained in these and see if we can expand access to them - to all of |
|
@rvagg I seem to remember there was a way to find the person who triggered a change. Can you remind me how? Looking in https://github.com/nodejs/jenkins-config-test/blob/master/jobs/libuv-test-commit.xml I've not found it yet. |
|
@mhdawson that job doesn't have it! It could be a newer feature and that job hasn't been touched since that feature was added. See https://github.com/nodejs/jenkins-config-test/blob/c9a2f917b204e76735419bdeb0ef6873c22d1c28/jobs/node-test-commit-linux.xml#L91, it's apparently a plugin, |
I edited that job 9 days ago (#2170 (comment)). Maybe Jenkins doesn't count changes to permissions when updating the last saved fields? |
|
As an aside, I've been using the backups to grep for references to machines we're planning to remove from the CI (#2273 and #2159) which has helped greatly. I've noticed that the backups don't appear to account for deleted jobs? e.g. for #2159 when I grepped for |
|
no, backup script's don't account for deletions .. that's been left as a TODO. But the scripts are in both of the repos (you should have access to the release one now too I think). See https://github.com/nodejs/jenkins-config-test/blob/master/backup.sh if you want to propose some changes. |
|
@richardlau can this be closed? Joao added the functionality for deleted jobs here https://github.com/nodejs/jenkins-config-test/commit/f90b5aa85935409e72bea78413e1ef5526d3a6d2 |
Yeah, I think we can close this and open new issues for any follow ups. |
Originally posted by @sam-github in #2087 (comment)
I think the question of tracking and documenting changes is an issue, but not directly related.
Maybe it should be in a seperate issue, but continuing here since its where we are: if Jenkins itself had a way of tracking and explaining config changes, that'd be ideal, though I haven't bumped into it (but I'm far from a Jenkins expert).
If its not a Jenkins featue, our options seem limited.
A github centric expectation would be that for any config change there should be an issue open that its related to, and perhaps there should be a
configlabel applied to every issue that resulted in or is related to a jenkins config change.I think this is pretty reasonable, but is not perfect:
git blame-like ability to look at something in the Jenkins UI, and then can somehow follow a trail back the github issue that motivated the changeThe text was updated successfully, but these errors were encountered: