-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add pipeline for flaky test runner job #46740
Conversation
💚 Build Succeeded |
💚 Build Succeeded |
This comment has been minimized.
This comment has been minimized.
5b4e38e
to
b86426a
Compare
Pinging @elastic/kibana-operations |
💚 Build Succeeded |
b86426a
to
c0e9c28
Compare
💔 Build Failed |
c0e9c28
to
903c898
Compare
💔 Build Failed |
retest |
💚 Build Succeeded |
903c898
to
65d26a8
Compare
|
||
def buildStatus = buildUtils.getBuildStatus() | ||
if (buildStatus != 'SUCCESS' && buildStatus != 'ABORTED') { | ||
node('flyweight') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did change this function a little bit so that a node isn't allocated unless an e-mail actually needs to go out... Otherwise, the code in this file is just moved over from the original Jenkinsfile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this something we could just run on master?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there actually a label set up for master? I didn't realize
💚 Build Succeeded |
💚 Build Succeeded |
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
This reverts commit e894255.
… into console-token-iterator * 'console-token-iterator' of github.com:jloleysens/kibana: (184 commits) [functional/services] update webdriver lib and types (elastic#47381) Standardizing IconField implementation across the app (elastic#47196) Move ui/value_suggestions ⇒ NP data plugin (elastic#45762) Remove ui/persisted_log - Part 2 (elastic#47236) Update gulp related packages (elastic#47421) Update dependency idx to ^2.5.6 (elastic#47399) try running fewer jobs in parallel on the same worker (elastic#47403) Update webpack related packages (elastic#47402) Update jsonwebtoken related packages (elastic#47400) Update gulp related packages (major) (elastic#46665) Update dependency prettier to ^1.18.2 (elastic#47340) Update dependency @types/puppeteer to ^1.20.1 (elastic#47339) Update dependency @elastic/elasticsearch to ^7.4.0 (elastic#47338) Update dependency tar-fs to ^1.16.3 (elastic#47341) [Code] Code Integrator Component (elastic#47180) [Canvas][i18n] Sidebar (elastic#46090) Generate uuid in task Manager as Kibana uuid may not yet have been initialised [Code] Embedded Code Snippet Component (elastic#47183) Revert "Add pipeline for flaky test runner job (elastic#46740)" SearchSource: fix docvalue_fields and fields intersection logic (elastic#46724) ...
This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
* Re-add pipeline for flaky test runner job (#48781) * Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable * Retry git clone up to 8 times before failing a build (#50734) * Flaky test runner improvements/fixes (#49367) * Allocate multiple flaky test runner agents as executions grow * WIP support for deleting kibana install dir during CI * Add setup script for testing scripts * Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner * Change flaky test runner worker processes from 8 to 12 * Add labels to shell scripts in Jenkins (#49657) * Change pipeline timeout from 3 hours to 2 hours (#51098) * Remove flaky pipeline step not relevant for 6.8 * Re-add failed_tests cli * Fix PARALLEL_PIPELINE_WORKER_INDEX var
Closes #46163
How to start the job: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/build
The job will execute the chosen test suite
NUMBER_EXECUTIONS
times, using 14 parallel processes on one machine. The build only happens once, then the suite is executed many times.Duplicate test results will show up, so if the same suite errors out 5 times, all 5 instances will show up.
Example: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4
Example of flaky test failing multiple times: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4/testReport/
End of console:
I was able to move a lot of Jenkinsfile code to a shared location, but it unfortunately required creating a
vars
directory in the root of the kibana repo. This is required by Jenkins shared libraries.