Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pipeline for flaky test runner job #46740

Merged
merged 22 commits into from
Oct 4, 2019

Conversation

brianseeders
Copy link
Contributor

@brianseeders brianseeders commented Sep 26, 2019

Closes #46163

How to start the job: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/build

The job will execute the chosen test suite NUMBER_EXECUTIONS times, using 14 parallel processes on one machine. The build only happens once, then the suite is executed many times.

Duplicate test results will show up, so if the same suite errors out 5 times, all 5 instances will show up.

Example: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4
Example of flaky test failing multiple times: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4/testReport/

Screen Shot 2019-09-27 at 2 02 21 PM

End of console:

11:47:39  There were 8 test suite failures.
11:47:39  [Pipeline] echo
11:47:39  The executions that failed were:
11:47:39  [Pipeline] echo
11:47:39  worker-14-0
11:47:39  worker-4-0
11:47:39  worker-4-1
11:47:39  worker-9-1
11:47:39  worker-2-1
11:47:39  worker-3-1
11:47:39  worker-13-1
11:47:39  worker-5-1
11:47:39  [Pipeline] echo
11:47:39  Please check 'Test Result' and 'Pipeline Steps' pages for more info

I was able to move a lot of Jenkinsfile code to a shared location, but it unfortunately required creating a vars directory in the root of the kibana repo. This is required by Jenkins shared libraries.

@brianseeders brianseeders added v8.0.0 Feature:CI Continuous integration labels Sep 26, 2019
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine

This comment has been minimized.

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

Jenkinsfile Outdated Show resolved Hide resolved
@brianseeders
Copy link
Contributor Author

retest

@snide snide removed the v7.4.0 label Oct 1, 2019
@elasticmachine
Copy link
Contributor

💚 Build Succeeded


def buildStatus = buildUtils.getBuildStatus()
if (buildStatus != 'SUCCESS' && buildStatus != 'ABORTED') {
node('flyweight') {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did change this function a little bit so that a node isn't allocated unless an e-mail actually needs to go out... Otherwise, the code in this file is just moved over from the original Jenkinsfile

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something we could just run on master?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there actually a label set up for master? I didn't realize

@brianseeders brianseeders marked this pull request as ready for review October 1, 2019 16:19
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@brianseeders brianseeders merged commit e894255 into elastic:master Oct 4, 2019
@brianseeders brianseeders deleted the flaky-test-runner branch October 4, 2019 17:53
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 4, 2019
* WIP Jenkinsfile for flaky test runner

* Fix syntax

* A few more jenkinsfile fixes

* A few more fixes

* Can't round numbers in Jenkins groovy sandbox apparently

* More fixes

* Only do build_kbn_tp_sample_panel_action once during flaky testing

* Fix path and try setting a different JOB value

* Keep track of flaky test failures

* Introduce flaky test for testing

* Some flaky test pipeline cleanup

* Fix a couple of issues with flaky test failure tracking

* Update flaky test runner build name/desc with metadata

* Revert "Introduce flaky test for testing"

This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32.

* Try adding a local shared library

* Move local library loading logic to kibana pipeline library

* Move shared groovy code to shared library

* Add missed file

* Add ability to specify multiple agents for flaky test runner

* Update kibana-pipeline-library version

* Fix bug causing early exit for oss ciGroup1
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 4, 2019
* WIP Jenkinsfile for flaky test runner

* Fix syntax

* A few more jenkinsfile fixes

* A few more fixes

* Can't round numbers in Jenkins groovy sandbox apparently

* More fixes

* Only do build_kbn_tp_sample_panel_action once during flaky testing

* Fix path and try setting a different JOB value

* Keep track of flaky test failures

* Introduce flaky test for testing

* Some flaky test pipeline cleanup

* Fix a couple of issues with flaky test failure tracking

* Update flaky test runner build name/desc with metadata

* Revert "Introduce flaky test for testing"

This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32.

* Try adding a local shared library

* Move local library loading logic to kibana pipeline library

* Move shared groovy code to shared library

* Add missed file

* Add ability to specify multiple agents for flaky test runner

* Update kibana-pipeline-library version

* Fix bug causing early exit for oss ciGroup1
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 4, 2019
* WIP Jenkinsfile for flaky test runner

* Fix syntax

* A few more jenkinsfile fixes

* A few more fixes

* Can't round numbers in Jenkins groovy sandbox apparently

* More fixes

* Only do build_kbn_tp_sample_panel_action once during flaky testing

* Fix path and try setting a different JOB value

* Keep track of flaky test failures

* Introduce flaky test for testing

* Some flaky test pipeline cleanup

* Fix a couple of issues with flaky test failure tracking

* Update flaky test runner build name/desc with metadata

* Revert "Introduce flaky test for testing"

This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32.

* Try adding a local shared library

* Move local library loading logic to kibana pipeline library

* Move shared groovy code to shared library

* Add missed file

* Add ability to specify multiple agents for flaky test runner

* Update kibana-pipeline-library version

* Fix bug causing early exit for oss ciGroup1
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 4, 2019
* WIP Jenkinsfile for flaky test runner

* Fix syntax

* A few more jenkinsfile fixes

* A few more fixes

* Can't round numbers in Jenkins groovy sandbox apparently

* More fixes

* Only do build_kbn_tp_sample_panel_action once during flaky testing

* Fix path and try setting a different JOB value

* Keep track of flaky test failures

* Introduce flaky test for testing

* Some flaky test pipeline cleanup

* Fix a couple of issues with flaky test failure tracking

* Update flaky test runner build name/desc with metadata

* Revert "Introduce flaky test for testing"

This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32.

* Try adding a local shared library

* Move local library loading logic to kibana pipeline library

* Move shared groovy code to shared library

* Add missed file

* Add ability to specify multiple agents for flaky test runner

* Update kibana-pipeline-library version

* Fix bug causing early exit for oss ciGroup1
brianseeders added a commit that referenced this pull request Oct 4, 2019
jloleysens added a commit to jloleysens/kibana that referenced this pull request Oct 7, 2019
… into console-token-iterator

* 'console-token-iterator' of github.com:jloleysens/kibana: (184 commits)
  [functional/services] update webdriver lib and types (elastic#47381)
  Standardizing IconField implementation across the app (elastic#47196)
  Move ui/value_suggestions ⇒ NP data plugin (elastic#45762)
  Remove ui/persisted_log - Part 2 (elastic#47236)
  Update gulp related packages (elastic#47421)
  Update dependency idx to ^2.5.6 (elastic#47399)
  try running fewer jobs in parallel on the same worker (elastic#47403)
  Update webpack related packages (elastic#47402)
  Update jsonwebtoken related packages (elastic#47400)
  Update gulp related packages (major) (elastic#46665)
  Update dependency prettier to ^1.18.2 (elastic#47340)
  Update dependency @types/puppeteer to ^1.20.1 (elastic#47339)
  Update dependency @elastic/elasticsearch to ^7.4.0 (elastic#47338)
  Update dependency tar-fs to ^1.16.3 (elastic#47341)
  [Code] Code Integrator Component (elastic#47180)
  [Canvas][i18n] Sidebar (elastic#46090)
  Generate uuid in task Manager as Kibana uuid may not yet have been initialised
  [Code] Embedded Code Snippet Component (elastic#47183)
  Revert "Add pipeline for flaky test runner job (elastic#46740)"
  SearchSource: fix docvalue_fields and fields intersection logic (elastic#46724)
  ...
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 22, 2019
This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened
brianseeders added a commit that referenced this pull request Oct 25, 2019
* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
spalger pushed a commit to spalger/kibana that referenced this pull request Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit that referenced this pull request Nov 15, 2019
* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit that referenced this pull request Nov 15, 2019
* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit that referenced this pull request Nov 15, 2019
* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit to brianseeders/kibana that referenced this pull request Nov 26, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable
brianseeders added a commit that referenced this pull request Dec 2, 2019
* Re-add pipeline for flaky test runner job (#48781)

* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable

* Retry git clone up to 8 times before failing a build (#50734)

* Flaky test runner improvements/fixes (#49367)

* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12

* Add labels to shell scripts in Jenkins (#49657)

* Change pipeline timeout from 3 hours to 2 hours (#51098)

* Remove flaky pipeline step not relevant for 6.8

* Re-add failed_tests cli

* Fix PARALLEL_PIPELINE_WORKER_INDEX var
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:CI Continuous integration release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v6.8.4 v7.3.2 v7.4.1 v7.5.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] Design/implement flaky test testing for pipelines
4 participants