[CI]: CI pipeline #54

v1v · 2020-06-12T15:19:04Z

What

Enable CI pipeline to build and push the docker images.

The push event works for:

everything (branches, tags, PRs) and will push them in https://docker.elastic.co:7000/observability-ci/golang-crossbuild:
on master based to the default location set in the Makefile.

Tasks

Create JJBB definition (see [JJBB] create job definition for the golang-crossbuild project beats#19162)

Test

REPOSITORY=docker.elastic.co/observability-ci make build

$ docker images | grep observability-ci            
docker.elastic.co/observability-ci/fpm                      1.11.0                 1fbf2b9b5555        2 minutes ago       416MB
docker.elastic.co/observability-ci/golang-crossbuild        1.14.4-main-debian8    51fbb901cd8a        2 minutes ago       1.39GB
docker.elastic.co/observability-ci/golang-crossbuild        1.14.4-main-debian7    3c5321b6f233        2 minutes ago       1.01GB
docker.elastic.co/observability-ci/golang-crossbuild        1.14.4-s390x           8064f3af14bd        2 minutes ago       807MB
docker.elastic.co/observability-ci/golang-crossbuild        1.14.4-ppc             57b58c057ff9        2 minutes ago       994MB
...

REPOSITORY=docker.elastic.co/observability-ci make push

echo ">> Pushing docker.elastic.co/observability-ci/golang-crossbuild:1.10.8-base"
>> Pushing docker.elastic.co/observability-ci/golang-crossbuild:1.10.8-base
The push refers to repository [docker.elastic.co/observability-ci/golang-crossbuild]
772921e6e3a7: Preparing 
041e855fb031: Preparing 
e142d45a5747: Preparing 
...

v1v · 2020-06-15T09:42:14Z

I need your review:

@urso , @andrewkroh , @elastic/observablt-robots

I'm afraid I'm not an admin of this particular repo therefore I cannot assign reviewers.

Jenkinsfile

Co-authored-by: cachedout <mike.place@elastic.co>

v1v · 2020-06-15T12:37:58Z

Makefile.common

@@ -1,4 +1,4 @@
-REPOSITORY := docker.elastic.co/beats-dev
+REPOSITORY ?= docker.elastic.co/beats-dev


This is the only change in the code to be able to set the repository where the docker images can be pushed, by default as used to be, but as we will be using the staging stage, then this will help to generate docker images on a PR basis which can be consumed for testing purposes.

mikemadden42

LGTM

andrewkroh · 2020-06-15T17:35:57Z

I'm afraid I'm not an admin of this particular repo therefore I cannot assign reviewers.

This repo is tied in with membership in https://github.com/orgs/elastic/teams/beats-contributors if you want to join.

andrewkroh · 2020-06-15T17:55:20Z

on tag based to the default location set in the Makefile.

This implies that this repo will need to start doing some tagging. As of now we've been pushing new images whenever master changed. And we really only changed master when new Go releases were available. I'm not against requiring tagging, but we should have a plan for it if we go this route (when do we tag, what do the tags mean, how do we convey the tag in the image metadata).

v1v · 2020-06-16T06:21:30Z

@andrewkroh , thanks for the details, I'm not really familiar with the release process for this project, I was just thinking the tag could be matching the go version, but there are certain questions as you stated that they are required to be addressed beforehand.

What do we do from here?

kuisathaverat · 2020-06-16T08:01:39Z

as Victor said, the most logical tag is the Go version. Using tags adds the possibility to compile with any images generated for a previous Go version in case we need it for some test, or we want to compile with several Go versions at some point.

andrewkroh · 2020-06-16T13:37:24Z

To be clear I was talking about git tagging, which is what I assume is meant in "on tag based to the default location". Any build of the repo produces images for multiple Go versions. Those images have docker tags containing the Go version and the cross-compiler architecture (see build tags).

Perhaps the path forward is to not introduce git tagging into the image push process at this time. And setup the job to push changes from master to the default location in the Makefile.

I think we can get the same ability to reproduce old builds if we add an additional docker tag (without git tags) that contains a time component (e.g. 1.10.8-main-20191124). Then if I need to exactly reproduce some old Beats build for whatever reason, we can use the time tagged docker images from the time of the original build. The images already contain docker labels following http://label-schema.org/rc1/ that contain this repo's commit ID in case we need to know the exact Dockerfiles that went into producing the images.

If we do want to use git tags to trigger the image pushes then we should

push an additional docker tag that incorporates the git tag (e.g. push both 1.14.1-main-debian7 and 1.14.1-main-debian7-vXX)
add the git tag to the image labels (org.label-schema.version = $git_tag)

v1v · 2020-06-16T14:09:13Z

To keep it simple, I'll remove the on tag based event to be master based. Then we can start a conversation in another thread to discuss how the release process looks like and what are the things, if any, that we could change.

…ature/ci * 'feature/ci' of github.com:v1v/golang-crossbuild: Apply suggestions from code review

v1v · 2020-06-16T14:12:33Z

c5cc234 is now the new behaviour.

NOTE: if we merge this particular PR then the automation to push those docker images will happen immediately.

v1v · 2020-06-16T14:56:38Z

I'll merge this particular PR as soon as the github setup is in place (webhooks and permissions)

elasticmachine · 2020-06-18T10:06:30Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #54 updated]
Start Time: 2020-06-23T11:19:26.454+0000
Duration: 84 min 57 sec

v1v · 2020-06-18T11:01:34Z

Alright, we got all the required webhooks and service accounts in place to run in the CI!

As soon as the build gets green then I'll merge it

v1v · 2020-06-23T11:22:21Z

For the record ead3be2 removed the issues with the push.* which it's not required anymore at all (for furtherh details please refers to the infra docs).

Already tested locally:

$ docker tag docker.elastic.co/observability-ci/fpm:1.11.0 docker.elastic.co/observability-ci/fpm:test                                                                       
$ docker push docker.elastic.co/observability-ci/fpm:test                                                                   
The push refers to repository [docker.elastic.co/observability-ci/fpm]
aa5033bd89aa: Pushed 
dc29c4af552e: Pushed 
0f7b4d7b24a5: Pushed 
7deb88e599ba: Pushed 
8354d5896557: Mounted from observability-ci/golang-crossbuild 
test: digest: sha256:1c8721bf3d8095878b02855a8c3d90a6c6d380de49c2483929e68323df487687 size: 1368

[CI]: CI pipeline

d4a91af

v1v mentioned this pull request Jun 12, 2020

[JJBB] create job definition for the golang-crossbuild project elastic/beats#19162

Merged

v1v added 2 commits June 12, 2020 16:30

ci: reuse workspace for pushing and refactor variables

596d9f9

ci: VERSION is not possible to change at the moment

544c610

v1v marked this pull request as ready for review June 12, 2020 16:18

andresrc added the [zube]: In Review label Jun 15, 2020

cachedout approved these changes Jun 15, 2020

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

Apply suggestions from code review

cd32638

Co-authored-by: cachedout <mike.place@elastic.co>

kuisathaverat approved these changes Jun 15, 2020

View reviewed changes

v1v commented Jun 15, 2020

View reviewed changes

mikemadden42 approved these changes Jun 15, 2020

View reviewed changes

v1v added 2 commits June 16, 2020 15:10

Publish automatically when there is build on master

c5cc234

Merge branch 'feature/ci' of github.com:v1v/golang-crossbuild into fe…

98112e5

…ature/ci * 'feature/ci' of github.com:v1v/golang-crossbuild: Apply suggestions from code review

andrewkroh approved these changes Jun 16, 2020

View reviewed changes

v1v self-assigned this Jun 18, 2020

v1v marked this pull request as draft June 18, 2020 10:29

v1v added 3 commits June 18, 2020 11:36

ci: install go

b643ff7

ci: more declarative based with the environment

0fa39bc

ci: use ubuntu label that matches the ubuntu-16 version

e3f95ce

v1v marked this pull request as ready for review June 18, 2020 11:01

v1v added 3 commits June 18, 2020 12:35

ci: use the right credentials

4ae0dd0

[ci] refactor

a4faae5

push.* is not required

ead3be2

v1v requested review from andrewkroh and mikemadden42 June 23, 2020 12:59

andrewkroh approved these changes Jun 23, 2020

View reviewed changes

v1v merged commit 4dda323 into elastic:master Jun 24, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Jun 24, 2020

v1v deleted the feature/ci branch June 24, 2020 07:09

This was referenced Jun 24, 2020

DOCS: add CI build badge #56

Merged

Automate publishing golang-crossbuild images for new Go versions #25

Closed

andresrc removed the [zube]: Done label Jun 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI]: CI pipeline #54

[CI]: CI pipeline #54

v1v commented Jun 12, 2020 •

edited

Loading

v1v commented Jun 15, 2020

v1v Jun 15, 2020

mikemadden42 left a comment

andrewkroh commented Jun 15, 2020

andrewkroh commented Jun 15, 2020

v1v commented Jun 16, 2020

kuisathaverat commented Jun 16, 2020 •

edited

Loading

andrewkroh commented Jun 16, 2020

v1v commented Jun 16, 2020

v1v commented Jun 16, 2020

v1v commented Jun 16, 2020

elasticmachine commented Jun 18, 2020 •

edited

Loading

Build stats

v1v commented Jun 18, 2020

v1v commented Jun 23, 2020

		@@ -1,4 +1,4 @@
		REPOSITORY := docker.elastic.co/beats-dev
		REPOSITORY ?= docker.elastic.co/beats-dev

[CI]: CI pipeline #54

[CI]: CI pipeline #54

Conversation

v1v commented Jun 12, 2020 • edited Loading

What

Tasks

Test

v1v commented Jun 15, 2020

v1v Jun 15, 2020

Choose a reason for hiding this comment

mikemadden42 left a comment

Choose a reason for hiding this comment

andrewkroh commented Jun 15, 2020

andrewkroh commented Jun 15, 2020

v1v commented Jun 16, 2020

kuisathaverat commented Jun 16, 2020 • edited Loading

andrewkroh commented Jun 16, 2020

v1v commented Jun 16, 2020

v1v commented Jun 16, 2020

v1v commented Jun 16, 2020

elasticmachine commented Jun 18, 2020 • edited Loading

💚 Build Succeeded

Build stats

v1v commented Jun 18, 2020

v1v commented Jun 23, 2020

v1v commented Jun 12, 2020 •

edited

Loading

kuisathaverat commented Jun 16, 2020 •

edited

Loading

elasticmachine commented Jun 18, 2020 •

edited

Loading