Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI]: retry with sleep #20065

Merged
merged 4 commits into from
Jul 21, 2020
Merged

[CI]: retry with sleep #20065

merged 4 commits into from
Jul 21, 2020

Conversation

v1v
Copy link
Member

@v1v v1v commented Jul 20, 2020

What does this PR do?

Use retry with sleep step and refactored the codecov step with more resiliance using the CI worker utils script.

Why is it important?

Avoid the issues when codecov is not reachable:

curl -sSLo codecov https://codecov.io/bash
14:49:05  curl: (7) Failed to connect to codecov.io port 443: Connection timed out

Issues

See codecov/codecov-python#158

@v1v v1v self-assigned this Jul 20, 2020
@v1v v1v added the automation label Jul 20, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 20, 2020
@v1v v1v added the Team:Automation Label for the Observability productivity team label Jul 20, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 20, 2020
@v1v v1v added ci needs_team Indicates that the issue/PR needs a Team:* label labels Jul 20, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 20, 2020
@botelastic
Copy link

botelastic bot commented Jul 20, 2020

This pull request doesn't have a Team:<team> label.

@v1v v1v requested a review from a team July 20, 2020 15:10
@v1v v1v marked this pull request as ready for review July 20, 2020 15:10
@@ -947,7 +947,9 @@ def withBeatsEnv(Map args = [:], Closure body) {
if (archive) {
archiveTestOutput(testResults: '**/build/TEST*.xml', artifacts: '**/build/TEST*.out')
}
reportCoverage()
catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE') {
sh(label: 'Report to Codecov', script: '.ci/scripts/report-codecov.sh auditbeat filebeat heartbeat journalbeat libbeat metricbeat packetbeat winlogbeat')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the codecov step from the library?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhm, that's a good point, my aim was to add more resilience to the 4xx http errors when curling the codecov... so I'll iterate a few times and see if we can simplify the pipeline to use the steps as you said

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just raised an issue #20080 to fix the the codecov issues in the pipeline since the credentials are not in place as far as I see, so for the time being I'll keep this PR as is and I'll work on the issue afterwards

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 20, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20065 updated]

  • Start Time: 2020-07-21T10:43:14.959+0000

  • Duration: 67 min 53 sec

Test stats 🧪

Test Results
Failed 1
Passed 17736
Skipped 1902
Total 19639

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat OSS Unit tests / test_process – test_system.Test

    • Age: 1
    • Duration: 1.45
    • Error Details: False is not true : fd not found in any process events

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest
    • Description: mage build unitTest

    • Duration: 6 min 55 sec

    • Start Time: 2020-07-21T11:06:21.990+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-21T11:49:37.388Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-21T11:49:37.388Z] + '[' -f winlogbeat/build/coverage/full.cov ']'
[2020-07-21T11:49:37.896Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats
[2020-07-21T11:49:38.203Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-21T11:49:38.216Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Lint
[2020-07-21T11:49:38.311Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-21T11:49:38.394Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-21T11:49:38.473Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-21T11:49:38.554Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-21T11:49:38.630Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-21T11:49:38.727Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-21T11:49:38.820Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-21T11:49:38.907Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-21T11:49:38.995Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-21T11:49:39.094Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-21T11:49:39.176Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-21T11:49:39.254Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-21T11:49:39.329Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Packetbeat-Linux
[2020-07-21T11:49:39.403Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-21T11:49:39.478Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-21T11:49:39.557Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-21T11:49:39.636Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-21T11:49:39.711Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-21T11:49:39.786Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-21T11:49:39.859Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-21T11:49:39.935Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-21T11:49:40.083Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-21T11:49:40.193Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-21T11:49:40.294Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-21T11:49:40.386Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-21T11:49:40.487Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-07-21T11:49:40.597Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-21T11:49:40.697Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-21T11:49:40.799Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-21T11:49:40.903Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-21T11:49:41.003Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-21T11:49:41.101Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-21T11:49:41.204Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-21T11:49:41.294Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-21T11:49:41.388Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-21T11:49:41.491Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Packetbeat-Windows
[2020-07-21T11:49:41.588Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-21T11:49:41.678Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Filebeat-oss
[2020-07-21T11:49:41.785Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-07-21T11:49:41.875Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Libbeat-oss
[2020-07-21T11:49:41.982Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-21T11:49:42.071Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-07-21T11:49:42.158Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-07-21T11:49:42.250Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-07-21T11:49:42.340Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-07-21T11:49:42.452Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-07-21T11:49:42.545Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-21T11:49:42.934Z] + cat
[2020-07-21T11:49:42.934Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-21T11:49:42.934Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-21T11:49:51.083Z] runbld>>> runbld started
[2020-07-21T11:49:51.083Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-21T11:49:52.023Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20065' in order of occurrence in the config (last value wins).
[2020-07-21T11:49:53.405Z] runbld>>> Debug logging enabled.
[2020-07-21T11:49:53.405Z] runbld>>> Storing result
[2020-07-21T11:49:53.668Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-21T11:49:53.668Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200721114953-BC7B6D1C
[2020-07-21T11:49:53.668Z] runbld>>> Adding system facts.
[2020-07-21T11:49:54.616Z] runbld>>> Adding vcs info for the latest commit:  a03fe70dd29bba8c60d2d9feb1e52e07ea2295a1
[2020-07-21T11:49:54.616Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-21T11:49:54.616Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-21T11:49:54.876Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-21T11:49:54.877Z] Processing JUnit reports with runbld...
[2020-07-21T11:49:55.137Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-21T11:49:55.137Z] runbld>>> DURATION: 31ms
[2020-07-21T11:49:55.137Z] runbld>>> STDOUT: 40 bytes
[2020-07-21T11:49:55.137Z] runbld>>> STDERR: 49 bytes
[2020-07-21T11:49:55.137Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-21T11:49:55.137Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats
[2020-07-21T11:49:56.522Z] runbld>>> Storing build metadata: 
[2020-07-21T11:49:56.522Z] runbld>>> Adding test report.
[2020-07-21T11:49:56.522Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats
[2020-07-21T11:49:57.093Z] runbld>>> Found 136 test output files
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-21T11:49:59.008Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20065/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-21T11:50:00.922Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 19489 Skipped: 1632
[2020-07-21T11:50:00.922Z] runbld>>> Storing result
[2020-07-21T11:50:00.922Z] runbld>>> FAILURES: 1
[2020-07-21T11:50:01.491Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-21T11:50:01.491Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200721114953-BC7B6D1C
[2020-07-21T11:50:01.491Z] runbld>>> Email notification disabled by environment variable.
[2020-07-21T11:50:01.491Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-21T11:50:06.996Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20065
[2020-07-21T11:50:07.100Z] [INFO] getVaultSecret: Getting secrets
[2020-07-21T11:50:07.173Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-21T11:50:07.949Z] + chmod 755 generate-build-data.sh
[2020-07-21T11:50:07.949Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20065/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20065/runs/3 FAILURE 4012731
[2020-07-21T11:50:07.949Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20065/runs/3/steps/?limit=10000 -o steps-info.json
[2020-07-21T11:50:10.432Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20065/runs/3/tests/?status=FAILED -o tests-errors.json
[2020-07-21T11:50:11.871Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20065/runs/3/log/ -o pipeline-log.txt

.ci/scripts/report-codecov.sh Outdated Show resolved Hide resolved
@v1v v1v merged commit cf0e2cf into elastic:master Jul 21, 2020
@v1v v1v deleted the feature/retry-with-sleep branch July 21, 2020 11:52
v1v added a commit to v1v/beats that referenced this pull request Jul 21, 2020
v1v added a commit to v1v/beats that referenced this pull request Jul 21, 2020
v1v added a commit that referenced this pull request Jul 21, 2020
v1v added a commit that referenced this pull request Jul 21, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automation ci Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants