Jenkins: Document Jenkins setup #32

grahamwhaley · 2018-06-01T14:12:12Z

We don't have an overview of how our Jenkins CI is hung
together. Add a document to both aid any new instances
being set up, and to help us if we ever need to re-create
the CI.

Fixes: #30

Signed-off-by: Graham whaley graham.whaley@intel.com

grahamwhaley · 2018-06-01T14:13:25Z

I expect this will need a touch more work - there is probably info I have missed and stuff that @chavafg might like to add.

/cc @gnawux @Pennyzct @Weichen81 @nitkon - let me know if this helps, and what it is not telling you ;-)

jodh-intel

Thanks for putting this together @grahamwhaley!

@klynnrif - ptal.

jodh-intel · 2018-06-01T14:15:09Z

Jenkins_setup.md

+Kata Containers CI operates two types of CI builds:
+
+- PR builds. Builds are scheduled upon submission or change to a Pull Request on any
+of the repositories


Missing period.

jodh-intel · 2018-06-01T14:15:36Z

Jenkins_setup.md

@@ -0,0 +1,161 @@
+# Kata Containers Jenkins CI setup


Could you add a table of contents for easy navigation?

jodh-intel · 2018-06-01T14:18:38Z

Jenkins_setup.md

+export ghprbTargetBranch
+
+cd $HOME
+git clone https://github.com/kata-containers/tests.git


I think we need to clone into the proper go path otherwise .ci/jenkins_job_build.sh will check itself out again to that path.

And I agree! - but, the above is how it is done today in our CI (in the QA CI anyhow - iirc, in the metrics CI I use a 'go get' if I can rather than a raw git clone).
/cc @chavafg @sboeuf for input. I will leave as is at present in this doc unless we change it in our live Jenkins.

jodh-intel · 2018-06-01T14:19:30Z

Jenkins_setup.md

+>
+> The `export` of the `ghprb` variables for the Master builds is benign, as the Master
+> builds do not use the GHPRB plugin. Subsequently, the `jenkins_job_build.sh` script can
+> still test for the GHPRB varibles to distinguish between a PR and Master build.


Typo: "variables".

jodh-intel · 2018-06-01T14:20:57Z

Jenkins_setup.md

+
+The teardown script is entered into the Jenkins Post-Build Actions dialog like:
+
+![Jenkins Build dialog](pictures/jenkins_build.png)


This is the same pic as the build script. Do we need a new image for the teardown?

heh, have the image already - did the cut/paste but forgot to update the link ;-) Fixed.

jodh-intel · 2018-06-01T14:21:15Z

Jenkins_setup.md

+
+| Var                 | Effect                                                           |
+| ---------           | ---------------------------------------------------------------- |
+| `WORKSPACE`         | Must be set. Is set by Jenkins to point at the directory the script will use as its workspace (scratch area) |


Bonus points for sorting the rows ;)

I can do - I sort of put them in the order the script evaluates them. I'll sort them, but if you think they might actually be better in order-of-use, ping back here.

jodh-intel · 2018-06-01T14:21:44Z

Jenkins_setup.md

+
+## Experienced users
+
+For the more experienced Jenkins user, if you are au-fait reading Jenkins xml config files


jodh-intel · 2018-06-01T14:23:22Z

Jenkins_setup.md

+then you can find a backup of our Azure based Jenkins QA CI in this repository under the
+[jenkins sub folder](https://github.com/kata-containers/ci/tree/master/jenkins).
+
+## Other CIs


This is correct info, but I'm not sure we need to include it? If we retain it, it may be worth explaining why we switched.

iirc, somebody mentioned they had CI hardware available for a non-x86 arch, but it was already using an alternative CI system (perhaps Travis) - so, I think maybe we leave this for the minute, and then get the other arch CIs up, and if we all end up on Jenkins or another CI, then we can drop it again... OK?

Happy to add a couple of details on why we moved - can you remember a couple of snippets? iirc, Travis does not support nested VMs (at least not the free-to-open-source version), and semaphore?? Did we just have speed/stability/reliability issues?

klynnrif

Scrubbed for grammar, structure, and flow :) Thanks!

klynnrif · 2018-06-01T16:27:37Z

Jenkins_setup.md

+# Kata Containers Jenkins CI setup
+
+The main Continuous Integration (CI) system for Kata Containers is [Jenkins](https://jenkins.io/).
+This document gives an overview of how Kata Containers integrates with Jenkins, and


Lines 4-6 read a little awkward to me - suggested rewrite: This document is an overview of how Kata Containers integrates with Jenkins. It also directs you to configuration information and scripts to help you set up a Kata Containers Jenkins CI system.

klynnrif · 2018-06-01T16:28:58Z

Jenkins_setup.md

+PR builds are triggered from Jenkins using the [GitHub Pull Request Builder](https://plugins.jenkins.io/ghprb) (GHPRB)
+plugin.
+
+The GHPRB plugin is configured to be triggered via github hooks. This is the preferred method


The GHPRB plugin is configured to trigger through GitHub hooks. ...

klynnrif · 2018-06-01T16:29:32Z

Jenkins_setup.md

+The GHPRB plugin is configured to be triggered via github hooks. This is the preferred method
+to trigger on changes (preferred to polling). The preferred method is to integrate the CIs
+using the [katacontainersbot](https://github.com/katacontainersbot) user. Please discuss
+your requirements with the Kata Containers team who can then discuss the appropriate changes.


your requirements with the Kata Containers team who will then discuss the appropriate changes.

klynnrif · 2018-06-01T16:30:27Z

Jenkins_setup.md

+using the [katacontainersbot](https://github.com/katacontainersbot) user. Please discuss
+your requirements with the Kata Containers team who can then discuss the appropriate changes.
+
+The GHPRB triggers should be set up similar to:


Set up the GHPRB triggers similar to the following:

klynnrif · 2018-06-01T16:31:06Z

Jenkins_setup.md

+The GHPRB triggers should be set up similar to:
+![Jenkins GHPRB plugin trigger config](pictures/ghprb_trigger.png)
+
+The GHPRB `Trigger Setup` further options should be set up similar to:


Set up the GHPRB Trigger Setup further options similar to the following:

klynnrif · 2018-06-01T16:42:09Z

Jenkins_setup.md

+| `WORKSPACE`         | Must be set. Is set by Jenkins to point at the directory the script will use as its workspace (scratch area) |
+| `ghprbTargetBranch` | Is set by the GHPRB Jenkins plugin. Is only set for PR builds, and is used with `ghprbPullId` to distinguish between PR and Master builds. |
+| `ghprbPullId`       | Is used in PR builds to locate the correct branch on github to pull and build. |
+| `KATA_DEV_MODE`     | Must not be set, or the script will not act as a CI script. KATA_DEV_MODE is a developer only option allowing the script to be run 'safely' in a development environment. |


Must not be set, or the script does not act as a CI script. ...

klynnrif · 2018-06-01T16:42:33Z

Jenkins_setup.md

+| `ghprbTargetBranch` | Is set by the GHPRB Jenkins plugin. Is only set for PR builds, and is used with `ghprbPullId` to distinguish between PR and Master builds. |
+| `ghprbPullId`       | Is used in PR builds to locate the correct branch on github to pull and build. |
+| `KATA_DEV_MODE`     | Must not be set, or the script will not act as a CI script. KATA_DEV_MODE is a developer only option allowing the script to be run 'safely' in a development environment. |
+| `METRICS_CI`        | If set, will skip running the QA tests after the install/build phases (as the tests are not required for a metrics run). |


If set, skips running the QA tests after the install/build phases ...

klynnrif · 2018-06-01T16:43:05Z

Jenkins_setup.md

+| `GOPATH`            | Is set to the installed Go environment                           |
+| `GOROOT`            | Is set to the installed Go environment                           |
+| `PATH`              | Is modified to include ${GOPATH}/bin, /usr/local/go/bin and the 'sbin' directories. |
+| `kata_repo`         | Is set to the github repo name, as passed in to $1 of the script. |


github => GitHub

klynnrif · 2018-06-01T16:45:51Z

Jenkins_setup.md

+## Other CIs
+
+Formerly Kata Containers, and one of its predecessors [Clear Containers](https://github.com/clearcontainers)
+have used both [Travis](https://travis-ci.com/) and [SemaphoreCI](https://semaphoreci.com/)


used both Travis and SemaphoreCI

klynnrif · 2018-06-01T16:48:27Z

Jenkins_setup.md

+
+## Experienced users
+
+For the more experienced Jenkins user, if you are au-fait reading Jenkins xml config files


Lines 151-152 suggested rewrite: For the more experienced Jenkins user, if you are comfortable reading Jenkins xml config files, there is a backup of our Azure-based Jenkins QA CI in this repository under the

grahamwhaley · 2018-06-04T11:13:48Z

@klynnrif @jodh-intel - updated and pushed - please re-review.

jodh-intel

Nice! Just a few nits and suggestions.

lgtm

jodh-intel · 2018-06-04T12:15:01Z

Jenkins_setup.md

+The Kata Containers Jenkins CI system uses Jenkins 'Freestyle Project' build jobs
+to perform its builds. Kata Containers CI operates two types of CI builds:
+
+- PR builds. Builds are scheduled upon submission or change to a Pull Request on any


Not essential, but if you change the layout to something like the following:

- [PR builds](#pr-builds) Builds are scheduled upon submission or change to a Pull Request on any of the repositories.

.. it'll render more clearly as:

PR builds

Builds are scheduled upon submission or change to a Pull Request on any of the repositories.

I also added the links to the sections.

jodh-intel · 2018-06-04T12:16:58Z

Jenkins_setup.md

+plugin.
+
+The GHPRB plugin is configured to trigger through GitHub hooks. This is the preferred method
+to trigger on changes (preferred to polling). The preferred method is to integrate the CIs


Nit: Third mention of the word "preferred" in two lines.

jodh-intel · 2018-06-04T12:19:09Z

Jenkins_setup.md

+The GHPRB plugin is configured to trigger through GitHub hooks. This is the preferred method
+to trigger on changes (preferred to polling). The preferred method is to integrate the CIs
+using the [katacontainersbot](https://github.com/katacontainersbot) user. Please discuss
+your requirements with the Kata Containers team who will then discuss the appropriate changes.


s/then discuss/then make/ ?

jodh-intel · 2018-06-04T12:20:09Z

Jenkins_setup.md

+using the [katacontainersbot](https://github.com/katacontainersbot) user. Please discuss
+your requirements with the Kata Containers team who will then discuss the appropriate changes.
+
+Set the GHPRB triggers similar to the following:


I'm wondering if you made the two "Set the ..." sentences to bullets whether that would auto-indent the graphics slightly for better clarity?

Ooh, testing MD changes on a branch you've already PR'd is a pain in the (fork/push) fingers :-( So, I went to have a look-see, and came across this just for you @jodh-intel :-) heh heh. The other solution involved embedded HTML, which we are not doing either... let me have a play with the bullets though...

That is ✨ deep magic ✨ indeed.

Bullets was the only way I could find to indent the images - but adding bullets to either the leading text or the image itself both looked a bit odd. I'll leave as is.

jodh-intel · 2018-06-04T12:21:14Z

Jenkins_setup.md

+### Master builds
+
+Master branch builds are tracked with the Jenkins [git Source Code Management](https://plugins.jenkins.io/git)
+plugin. GITScm GitHub hooks are used to trigger builds.


It's pretty clear but for consistency with other docs, maybe you could take " (GITScm)" to the end of the first sentence before using that term?

jodh-intel · 2018-06-04T12:22:40Z

Jenkins_setup.md

+
+| Var                 | Effect                                                           |
+| ---------           | ---------------------------------------------------------------- |
+| `CI`                | Is set by the script (unless KATA_DEV_MODE is already set) to indicate to sub-scripts that this is a CI build. |


KATA_DEV_MODE.

This is the output var section - the script doesn't modify or set KATA_DEV_MODE does it?

oh - maybe the comment hasn't lined up with the correct table row. I just meant to render that variable in a fixed font:

s/unless KATA_DEV_MODE is/unless KATA_DEV_MODE is/

Ah, I see - got it now. Fixed, along with the one in the previous table as well.

jodh-intel · 2018-06-04T12:23:46Z

Jenkins_setup.md

+
+## Experienced users
+
+For the more experienced Jenkins user, if you are au-fait reading Jenkins XML config files


Italics for au-fait as it's a foreign language term. I wonder if we should replace for simplicity though?

s/au-fait/comfortable/ ?

spoil sport.... I guess I'll change it ;-)

grahamwhaley · 2018-06-04T13:02:00Z

Hi @jongwu.

hi @jodh-intel @grahamwhaley , where does jenkins_job_build.sh run into, in the $JENKINS_HOME or in jenkins web configure UI?
I think the steps to set up jenkins ci system in a new machine is : 1. install jenkins 2. add token and credential to connect github 3 add jobs. is that right? when to run jenkins_job_build.sh?

I think you have now figured where the script is run from - in the Build/Execute Shell section of the Jenkins FreeStyle job setup dialog?
I updated this PR a little earlier. Your list is pretty accurate. Maybe I would add an extra step:

Install Jenkins
Install required plugins (GHPRB for instance)
Add github token (or you can do this later when configuring the FreeStyle project iirc)
Create and configure a Freestyle job

If is the first time integrating Jenkins with Kata Containers then it is probably most useful to start with a single 'Master' job that is tied to the runtime repository, and then use the 'Build now' option so you can check builds are working before hooking up a live PR tracking job :-)

grahamwhaley · 2018-06-04T15:15:53Z

Updates applied @jodh-intel

jodh-intel · 2018-06-04T15:23:09Z

Jenkins_setup.md

+| `GOPATH`            | Is set to the installed Go environment                           |
+| `GOROOT`            | Is set to the installed Go environment                           |
+| `kata_repo`         | Is set to the GitHub repo name, as passed in to $1 of the script. |
+| `PATH`              | Is modified to include ${GOPATH}/bin, /usr/local/go/bin and the 'sbin' directories. |


Nit: could use backticks for these.

jodh-intel · 2018-06-04T15:23:18Z

lgtm

chavafg

lgtm

This is very well documented. Thanks @grahamwhaley

klynnrif

I missed one instance of GitHub with a lowercase h. That is the only additional change. Thanks!

klynnrif · 2018-06-05T15:26:54Z

Jenkins_setup.md

+![Jenkins master setup](pictures/master_git_config.png)
+
+
+### Github bandwidth and tokens


Apologies if I missed this before: Github => GitHub

Fixed @klynnrif , also in the ToC :-)

Thank you!!!

klynnrif

lgtm - thanks!

jodh-intel · 2018-06-06T08:15:56Z

So close... just one more ack needed. /me look at @kata-containers/ci ... ;)

We don't have an overview of how our Jenkins CI is hung together. Add a document to both aid any new instances being set up, and to help us if we ever need to re-create the CI. Fixes: kata-containers#30 Signed-off-by: Graham whaley <graham.whaley@intel.com>

grahamwhaley · 2018-06-07T08:57:23Z

rebased to clear .gitconfig clash - and now we have the big green button.... anybody

jodh-intel · 2018-06-07T09:10:55Z

Looks like we need 1 more ack?

chavafg · 2018-06-23T02:08:19Z

lgtm

docs: Add a README

Apply the doc teams review changes that got missed off PR kata-containers#32. Fixes kata-containers#34. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

grahamwhaley requested a review from chavafg June 1, 2018 14:13

jodh-intel reviewed Jun 1, 2018

View reviewed changes

klynnrif suggested changes Jun 1, 2018

View reviewed changes

grahamwhaley force-pushed the 20180601_jenkins_info branch from 2570934 to 9cb006e Compare June 1, 2018 16:50

jodh-intel mentioned this pull request Jun 4, 2018

ARM: get the ARM CIs up #30

Closed

grahamwhaley force-pushed the 20180601_jenkins_info branch from 9cb006e to b1b9c20 Compare June 4, 2018 11:13

jodh-intel approved these changes Jun 4, 2018

View reviewed changes

grahamwhaley force-pushed the 20180601_jenkins_info branch from b1b9c20 to 2edbc57 Compare June 4, 2018 15:15

jodh-intel reviewed Jun 4, 2018

View reviewed changes

grahamwhaley force-pushed the 20180601_jenkins_info branch from 2edbc57 to 9792319 Compare June 4, 2018 15:41

chavafg approved these changes Jun 5, 2018

View reviewed changes

klynnrif suggested changes Jun 5, 2018

View reviewed changes

grahamwhaley force-pushed the 20180601_jenkins_info branch from 9792319 to 0f97abb Compare June 5, 2018 16:25

klynnrif approved these changes Jun 5, 2018

View reviewed changes

grahamwhaley force-pushed the 20180601_jenkins_info branch from 0f97abb to 4c5f8e1 Compare June 7, 2018 08:56

chavafg merged commit 454e60a into kata-containers:master Jun 23, 2018

GabyCT pushed a commit to GabyCT/ci that referenced this pull request Feb 12, 2019

Merge pull request kata-containers#32 from jodh-intel/add-readme

9c8e59d

docs: Add a README

GabyCT pushed a commit to GabyCT/ci that referenced this pull request Feb 12, 2019

docs: Improve README

e430047

Apply the doc teams review changes that got missed off PR kata-containers#32. Fixes kata-containers#34. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>


		The teardown script is entered into the Jenkins Post-Build Actions dialog like:

		![Jenkins Build dialog](pictures/jenkins_build.png)


		## Experienced users

		For the more experienced Jenkins user, if you are au-fait reading Jenkins xml config files

		![Jenkins master setup](pictures/master_git_config.png)


		### Github bandwidth and tokens

Jenkins: Document Jenkins setup #32

Jenkins: Document Jenkins setup #32

Conversation

grahamwhaley commented Jun 1, 2018

grahamwhaley commented Jun 1, 2018

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klynnrif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamwhaley commented Jun 4, 2018

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamwhaley commented Jun 4, 2018

grahamwhaley commented Jun 4, 2018

Choose a reason for hiding this comment

jodh-intel commented Jun 4, 2018

chavafg left a comment

Choose a reason for hiding this comment

klynnrif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klynnrif left a comment

Choose a reason for hiding this comment

jodh-intel commented Jun 6, 2018

grahamwhaley commented Jun 7, 2018

jodh-intel commented Jun 7, 2018

chavafg commented Jun 23, 2018