Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick to 7.9: packaging backports #22078

Conversation

mdelapenya
Copy link
Contributor

Backports the following commits to 7.9:

mdelapenya and others added 9 commits October 22, 2020 12:05
…lastic#21100)

* feat: add a new step to run the e2e tests for certain parts of Beats

We are going to trigger the tests for those parts affected by the
elastic-agent, filebeat, or metricbeat, because those are the ones we
verify  in the e2e-testing suite

* chore: do not include heartbeat

* feat: trigger the e2e tests

* fix: use relative path

* chore: use proper target branch name for PRs

* chore: use different tag

* fix: use proper env variable

* chore: pass github checks context to downstream job

* chore: revert shared lib version

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: add BASE_DIR env variable

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: remove duplicated env

* ffix: add param comma separator

* fix: wrong copy&paste

* chore: move e2e GH check out of the release context

* chore: simplify conditional logic

* chore: refine execution of test suites

* fix: use proper parameter name

* chore: set metricbeat version

* chore: remove slack notifications on PRs

* chore: update parameter

* chore: run multiple test suites per beat type

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
* fix: set versions ony for PRs

We want to use default versions per branch when running after a merge

* fix: add trailing comma

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
* fix: push ubi8 images too

* chore: enhance retries

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: use variables in log

* chore: add "-oss" images

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
* fix: remove extra curly brace

* chore: proper indent
It will create the artifacts with some requirements related to integrity
* chore: pass beat as a method argument (no side-effects)

* chore: run tests in a separate stage

* fix: use parenthesis

* chore: update comment

* chore: do not trigger E2E tests if no suite was added

* fix: use missing curly brackets

* fix: wrong closure wrapping

* fix: condition was not set
* fix: delegate pushes to variants

* chore: group conditions for x-pack

* chore: simplify with endsWith

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
# Conflicts:
#	.ci/packaging.groovy
* feat: push aliases for docker images

* feat: build alias for snapshots

* fix: only update alias on snapshots

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

* fix: wrong image name for alias

* fix: reuse variable as groovy does not hide variables by scope

* chore: extract common logic to a method

* Revert "fix: only update alias on snapshots"

This reverts commit cff2cef.

* Revert "feat: build alias for snapshots"

This reverts commit 707e0d7.

* chore: do not push aliases for PRs

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
@mdelapenya mdelapenya self-assigned this Oct 22, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 22, 2020
@mdelapenya mdelapenya requested review from v1v and kuisathaverat and removed request for v1v October 22, 2020 10:08
@mdelapenya mdelapenya added the Team:Automation Label for the Observability productivity team label Oct 22, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 22, 2020
@mdelapenya mdelapenya changed the title Cherry-pick to 7.9: feat: add a new step to run the e2e tests for certain parts of Beats (#21100) | [E2E Tests] fix: set versions ony for PRs (#21608) | [CI: Packaging] fix: push ubi8 images too (#21621) | fix: remove extra curly brace in s Cherry-pick to 7.9: packaging backports Oct 22, 2020
.ci/packaging.groovy Outdated Show resolved Hide resolved
@@ -149,74 +159,174 @@ pipeline {
}
}
}
stage('Run E2E Tests for Packages'){
agent { label 'ubuntu && immutable' }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
agent { label 'ubuntu && immutable' }
agent { label 'ubuntu-18 && immutable' }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we apply this to master and then backport?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have to use fixed Ubuntu workers version, it not it will run on 14.04, 16.04, 18.04, or 20.04 with a random result.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 22, 2020

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #22078 updated]

  • Start Time: 2020-10-22T10:15:48.239+0000

  • Duration: 16 min 56 sec

Steps errors 1

Expand to view the steps failures

  • Name: make check
    • Description: make check

    • Duration: 3 min 2 sec

    • Start Time: 2020-10-22T10:28:06.075+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-10-22T10:28:06.066Z]   TIMEOUT: null
[2020-10-22T10:28:06.066Z]   USERPROFILE: null
[2020-10-22T10:28:06.066Z]   VENV_PARAMS: null
[2020-10-22T10:28:06.066Z]   XPACK_SUFFIX: null
[2020-10-22T10:28:06.066Z]   ### END ENV DUMP ###
[2020-10-22T10:28:06.066Z]   
[2020-10-22T10:28:06.359Z] + make check
[2020-10-22T10:28:16.359Z] make[1]: Entering directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/libbeat'
[2020-10-22T10:28:24.497Z] >> fmt - go-licenser: Checking for missing headers
[2020-10-22T10:28:28.785Z] >> fmt - go-licenser: Adding missing headers
[2020-10-22T10:28:28.785Z] >> fmt - goimports: Formatting Go code
[2020-10-22T10:28:28.785Z] >> fmt - autopep8: Formatting Python code
[2020-10-22T10:28:50.752Z] >> check: Checking source code for common problems
[2020-10-22T10:30:12.293Z] make[1]: Leaving directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/libbeat'
[2020-10-22T10:30:12.293Z] make[1]: Entering directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/auditbeat'
[2020-10-22T10:30:15.616Z] >> fmt - go-licenser: Checking for missing headers
[2020-10-22T10:30:19.822Z] >> fmt - go-licenser: Adding missing headers
[2020-10-22T10:30:20.083Z] >> fmt - autopep8: Formatting Python code
[2020-10-22T10:30:20.083Z] >> fmt - goimports: Formatting Go code
[2020-10-22T10:30:23.489Z] Generated fields.yml for auditbeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/auditbeat/fields.yml
[2020-10-22T10:30:24.061Z] Generated fields.yml for auditbeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/auditbeat/fields.yml
[2020-10-22T10:30:24.061Z] >> Building auditbeat.yml for linux/amd64
[2020-10-22T10:30:24.061Z] >> Building auditbeat.reference.yml for linux/amd64
[2020-10-22T10:30:24.061Z] >> Building auditbeat.docker.yml for linux/amd64
[2020-10-22T10:30:26.610Z] exec: go list -m
[2020-10-22T10:30:27.182Z] Generated fields.yml for auditbeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/auditbeat/build/fields/fields.all.yml
[2020-10-22T10:30:28.125Z] >> check: Checking source code for common problems
[2020-10-22T10:30:32.348Z] make[1]: Leaving directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/auditbeat'
[2020-10-22T10:30:32.348Z] make[1]: Entering directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/filebeat'
[2020-10-22T10:30:40.501Z] >> fmt - go-licenser: Checking for missing headers
[2020-10-22T10:30:43.810Z] >> fmt - go-licenser: Adding missing headers
[2020-10-22T10:30:44.073Z] >> fmt - goimports: Formatting Go code
[2020-10-22T10:30:44.073Z] >> fmt - autopep8: Formatting Python code
[2020-10-22T10:30:48.288Z] No fields files for module apache2
[2020-10-22T10:30:48.289Z] Generated fields.yml for filebeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/filebeat/fields.yml
[2020-10-22T10:30:48.885Z] Generated fields.yml for filebeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/filebeat/fields.yml
[2020-10-22T10:30:48.885Z] >> Building filebeat.yml for linux/amd64
[2020-10-22T10:30:48.885Z] >> Building filebeat.reference.yml for linux/amd64
[2020-10-22T10:30:48.885Z] >> Building filebeat.docker.yml for linux/amd64
[2020-10-22T10:30:51.432Z] exec: go list -m
[2020-10-22T10:30:52.815Z] Generated fields.yml for filebeat to /var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/filebeat/build/fields/fields.all.yml
[2020-10-22T10:31:02.818Z] >> check: Checking source code for common problems
[2020-10-22T10:31:08.130Z] Error: some files are not up-to-date. Run 'mage fmt update' then review and commit the changes. Modified: [filebeat/modules.d/iis.yml.disabled]
[2020-10-22T10:31:08.130Z] ../libbeat/scripts/Makefile:150: recipe for target 'check' failed
[2020-10-22T10:31:08.130Z] make[1]: *** [check] Error 1
[2020-10-22T10:31:08.130Z] make[1]: Leaving directory '/var/lib/jenkins/workspace/Beats_beats_PR-22078/src/github.com/elastic/beats/filebeat'
[2020-10-22T10:31:08.130Z] Makefile:98: recipe for target 'check' failed
[2020-10-22T10:31:08.130Z] make: *** [check] Error 1
[2020-10-22T10:31:08.456Z] Client: Docker Engine - Community
[2020-10-22T10:31:08.456Z]  Version:           19.03.13
[2020-10-22T10:31:08.456Z]  API version:       1.40
[2020-10-22T10:31:08.456Z]  Go version:        go1.13.15
[2020-10-22T10:31:08.456Z]  Git commit:        4484c46d9d
[2020-10-22T10:31:08.456Z]  Built:             Wed Sep 16 17:02:36 2020
[2020-10-22T10:31:08.456Z]  OS/Arch:           linux/amd64
[2020-10-22T10:31:08.456Z]  Experimental:      false
[2020-10-22T10:31:08.456Z] 
[2020-10-22T10:31:08.456Z] Server: Docker Engine - Community
[2020-10-22T10:31:08.456Z]  Engine:
[2020-10-22T10:31:08.456Z]   Version:          19.03.13
[2020-10-22T10:31:08.456Z]   API version:      1.40 (minimum version 1.12)
[2020-10-22T10:31:08.456Z]   Go version:       go1.13.15
[2020-10-22T10:31:08.456Z]   Git commit:       4484c46d9d
[2020-10-22T10:31:08.456Z]   Built:            Wed Sep 16 17:01:06 2020
[2020-10-22T10:31:08.456Z]   OS/Arch:          linux/amd64
[2020-10-22T10:31:08.456Z]   Experimental:     false
[2020-10-22T10:31:08.456Z]  containerd:
[2020-10-22T10:31:08.456Z]   Version:          1.3.7
[2020-10-22T10:31:08.456Z]   GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
[2020-10-22T10:31:08.456Z]  runc:
[2020-10-22T10:31:08.456Z]   Version:          1.0.0-rc10
[2020-10-22T10:31:08.456Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-10-22T10:31:08.456Z]  docker-init:
[2020-10-22T10:31:08.456Z]   Version:          0.18.0
[2020-10-22T10:31:08.456Z]   GitCommit:        fec3683
[2020-10-22T10:31:08.456Z] Unable to find image 'alpine:3.4' locally
[2020-10-22T10:31:09.027Z] 3.4: Pulling from library/alpine
[2020-10-22T10:31:09.288Z] c1e54eec4b57: Pulling fs layer
[2020-10-22T10:31:09.548Z] c1e54eec4b57: Download complete
[2020-10-22T10:31:09.813Z] c1e54eec4b57: Pull complete
[2020-10-22T10:31:09.813Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-10-22T10:31:09.813Z] Status: Downloaded newer image for alpine:3.4
[2020-10-22T10:31:14.460Z] Stage "Build&Test" skipped due to earlier failure(s)
[2020-10-22T10:31:19.707Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-22T10:31:19.716Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-22T10:31:19.766Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-22078-2/source/source.tgz
[2020-10-22T10:31:19.793Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-22078-2/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-22078/source.tgz
[2020-10-22T10:31:30.902Z] + tar --version
[2020-10-22T10:31:31.203Z] + tar -xpf source.tgz
[2020-10-22T10:31:43.763Z] + rm source.tgz
[2020-10-22T10:31:43.822Z] [WARN] runbld: stashedTestReports is empty.
[2020-10-22T10:31:44.033Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-22078
[2020-10-22T10:31:44.102Z] [INFO] getVaultSecret: Getting secrets
[2020-10-22T10:31:44.153Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-22T10:31:44.771Z] + chmod 755 generate-build-data.sh
[2020-10-22T10:31:44.771Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22078/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22078/runs/2 FAILURE 956274
[2020-10-22T10:31:44.771Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22078/runs/2/steps/?limit=10000 -o steps-info.json
[2020-10-22T10:31:45.322Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22078/runs/2/tests/?status=FAILED -o tests-errors.json
[2020-10-22T10:31:45.322Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2020-10-22T10:31:46.665Z] Retry 2/3 exited 22, retrying in 2 seconds...

@mdelapenya
Copy link
Contributor Author

Should we merge this even though the 7.9 base branch's lint is broken?

@kuisathaverat
Copy link
Contributor

I really want to rid of make update and make check targets causes more problems than resolve

@kuisathaverat
Copy link
Contributor

I will see want happens

@kuisathaverat
Copy link
Contributor

kuisathaverat commented Oct 22, 2020

this PR break the check #21919, basically, it adds a new line to a file at the end, this breaks the make check I dunno where make check obtain the source to compare the files but it looks like does work well, I guess it generates some files and compares them to the files in the repo, but if we have to modify those files manually this comparison has no sense, also add a newline at the end of files is a good practice.

@mdelapenya mdelapenya merged commit 6c47e12 into elastic:7.9 Oct 22, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
* feat: add a new step to run the e2e tests for certain parts of Beats (elastic#21100)

* feat: add a new step to run the e2e tests for certain parts of Beats

We are going to trigger the tests for those parts affected by the
elastic-agent, filebeat, or metricbeat, because those are the ones we
verify  in the e2e-testing suite

* chore: do not include heartbeat

* feat: trigger the e2e tests

* fix: use relative path

* chore: use proper target branch name for PRs

* chore: use different tag

* fix: use proper env variable

* chore: pass github checks context to downstream job

* chore: revert shared lib version

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: add BASE_DIR env variable

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: remove duplicated env

* ffix: add param comma separator

* fix: wrong copy&paste

* chore: move e2e GH check out of the release context

* chore: simplify conditional logic

* chore: refine execution of test suites

* fix: use proper parameter name

* chore: set metricbeat version

* chore: remove slack notifications on PRs

* chore: update parameter

* chore: run multiple test suites per beat type

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* [E2E Tests] fix: set versions ony for PRs (elastic#21608)

* fix: set versions ony for PRs

We want to use default versions per branch when running after a merge

* fix: add trailing comma

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* [CI: Packaging] fix: push ubi8 images too (elastic#21621)

* fix: push ubi8 images too

* chore: enhance retries

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: use variables in log

* chore: add "-oss" images

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* fix: remove extra curly brace in script (elastic#21692)

* fix: remove extra curly brace

* chore: proper indent

* fix: update fleet test suite name (elastic#21738)

* chore: create CI artifacts for DEV usage (elastic#21645)

It will create the artifacts with some requirements related to integrity

* chore: simplify triggering the E2E tests for Beats (elastic#21790)

* chore: pass beat as a method argument (no side-effects)

* chore: run tests in a separate stage

* fix: use parenthesis

* chore: update comment

* chore: do not trigger E2E tests if no suite was added

* fix: use missing curly brackets

* fix: wrong closure wrapping

* fix: condition was not set

* chore: delegate variant pushes to the right method (elastic#21861)

* fix: delegate pushes to variants

* chore: group conditions for x-pack

* chore: simplify with endsWith

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
# Conflicts:
#	.ci/packaging.groovy

* feat: package aliases for snapshots (elastic#21960)

* feat: push aliases for docker images

* feat: build alias for snapshots

* fix: only update alias on snapshots

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

* fix: wrong image name for alias

* fix: reuse variable as groovy does not hide variables by scope

* chore: extract common logic to a method

* Revert "fix: only update alias on snapshots"

This reverts commit cff2cef.

* Revert "feat: build alias for snapshots"

This reverts commit 707e0d7.

* chore: do not push aliases for PRs

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

* chore: Use third number as x

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants