Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test runner improvements/fixes #49367

Merged

Conversation

brianseeders
Copy link
Contributor

@brianseeders brianseeders commented Oct 25, 2019

  • Scale agents up every 24 executions, up to 3 agents max
  • Remove kibana install dir at the end of each execution to free up space
  • Move a lot of duplicated setup code from all test runner scripts to test/scripts/jenkins_test_setup.sh

Removing kibana install dirs is pretty much required for the flaky test runner to work correctly, now that we're mounting the workspace in memory

@brianseeders brianseeders added Team:Operations Team label for Operations Team v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:CI Continuous integration v7.5.0 v7.6.0 v6.8.4 v7.4.2 labels Oct 25, 2019
@brianseeders brianseeders requested review from spalger and a team October 25, 2019 17:18
@brianseeders brianseeders self-assigned this Oct 25, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

.ci/Jenkinsfile_flaky Outdated Show resolved Hide resolved
Copy link
Member

@wayneseymour wayneseymour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome to remove more duplication!

test/scripts/jenkins_test_setup.sh Show resolved Hide resolved
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

for(def agentNumber = 1; agentNumber <= agentCount; agentNumber++) {
for(def agentNumber = 1; agentNumber <= AGENT_COUNT; agentNumber++) {
def agentNumberInside = agentNumber
def agentExecutions = floor(EXECUTIONS/AGENT_COUNT) + (agentNumber <= EXECUTIONS%AGENT_COUNT ? 1 : 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be better as a closure or method with a name?
No big deal, but names are quicker to grok, vs not being able to be lazy and having to read the code lol ahahahaa

Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brianseeders brianseeders merged commit 6468e79 into elastic:master Oct 30, 2019
@brianseeders brianseeders deleted the flaky-test-runner-improvements branch October 30, 2019 21:24
brianseeders added a commit to brianseeders/kibana that referenced this pull request Nov 26, 2019
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Dec 2, 2019
* Re-add pipeline for flaky test runner job (#48781)

* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable

* Retry git clone up to 8 times before failing a build (#50734)

* Flaky test runner improvements/fixes (#49367)

* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12

* Add labels to shell scripts in Jenkins (#49657)

* Change pipeline timeout from 3 hours to 2 hours (#51098)

* Remove flaky pipeline step not relevant for 6.8

* Re-add failed_tests cli

* Fix PARALLEL_PIPELINE_WORKER_INDEX var
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:CI Continuous integration release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v6.8.4 v7.4.2 v7.5.0 v7.6.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants