[ftr] automatically determine config run order #130983
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spalger
force-pushed
the
implement/ftr-config-run-order
branch
7 times, most recently
from
8589c46
to
8d0ba79
Compare
spalger
force-pushed
the
implement/ftr-config-run-order
branch
from
d4e4153
to
8bc1088
Compare
efegurkan
approved these changes
May 4, 2022
@kertal already updated and it switches from ciGroups to FTR Configs based on the contents of the PR under test. Try putting this PR url into the form! |
pheyos
approved these changes
May 4, 2022
brianseeders
approved these changes
May 4, 2022
thx @spencer, works ❤️🔥 |
added 4 commits
May 4, 2022 10:58
liza-mae
approved these changes
May 4, 2022
nreese
approved these changes
May 4, 2022
💛 Build succeeded, but was flakyFailed CI StepsMetrics [docs]Public APIs missing comments
Any counts in public APIs
History
To update your PR or re-run it, just comment with: |
|
It this point I think we have enough reviews to move forward. Thanks to everyone who took the time to review! |
kertal
pushed a commit
to kertal/kibana
that referenced
this pull request
May 24, 2022
* [ftr] automatically determine config run order * split lens config into two groups * support ftr configs always running against CI * Split detection_engine_api_integration rule exception list tests * Add configs from previous commit * [ftr] remove testMetadata and maintain a unique lifecycle instance per run * Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run" This reverts commit d2b4fdb. * Split alerting_api_integration/security_and_spaces tests * Add groups to yaml * Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"" This reverts commit 56232ee. * stop ES more forcefully and fix timeout * only cleanup lifecycle phases when the cleanup is totally complete * only use kill when cleaning up an esTestInstance * fix broken import * fix runOptions.alwaysUseSource implementation * fix config access * fix x-pack/ccs config * fix ml import file paths * update kibana build id * revert array.concat() change * fix baseConfig usage * fix pie chart data * split up maps tests * pull in all of group5 so that es archives are loaded correctly * add to ftr configs.yml * fix pie chart data without breaking legacy version * fix more pie_chart stuff in new vis lib * restore normal PR tasks * bump kibana-buildkite-library * remove ciGroup validation * remove the script which is no longer called from checks.sh * [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm' * adapt flaky test runner scripts to handle ftrConfig paths * fix types in alerting_api_integration * improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh * Split xpack dashboard tests * Add configs * [flaky] remove key from ftr-config steps * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * restore cypress builds * remove ciGroups from FTR config files * fixup some docs * add temporary script to hunt for FTR config files * use config.base.js naming for clarity * use script to power ftr_configs.yml * remove usage of removed x-pack/scripts/functional_tests * fix test names in dashboard snapshots * bump kibana-buildkite-library * Try retrying only failed configs * be a little quieter about trying to get testStats from configs with testRunners defined * Remove test code * bump kibana-buildkite-library * update es_snapshot and on_merge jobs too * track duration and exit code for each config and print it at the end of the script * store results in order, rather than by key, in case there are duplicates in $config * bash is hard * fix env source and use +e rather than disabling e for whole file * bash sucks * print config summary in jest jobs too * define results in jest_parallel.sh * simplify config summary print, format times a little better * fix reference to unbound time variable, use better variable name * skip the newline between each result * finish with the nitpicking * sync changes with ftr_configs.sh * refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml * fix config.edge.js base config import paths * fix some readmes * resolve paths from ftr_configs manifest * fix readConfigFile tests * just allow __fixtures__ configs * list a few more cypress config files * install the main branch of kibana-buildkite-library * split up lens group1 * move ml data_visualizer tests to their own config * fix import paths * fix more imports * install specific commit of buildkite-pipeline-library * sort configs in ftr_configs.yml * bump kibana-buildkite-library * remove temporary script * fix env var for limiting config types * Update docs/developer/contributing/development-functional-tests.asciidoc Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co> * produce a JUnit report for saved objects field count * apply standard concurrency limits from flaky test runner * support customizing FTR concurrency via the env Co-authored-by: Brian Seeders <brian.seeders@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on the work done in #130857 to run Functional Test Runner config files in workers using an order defined by the ci-stats API. This order is defined by searching through "TestGroup" records stored in ci-stats, and pulling in the most recent execution time for a specific config file, then grouping the config files together to make groups which should take less than the
maxtime passed to the API.This new strategy has a few impacts on the way that our functional tests are written:
ciGroup*tags are no longer needed and were therefore removedFTR Configs which take longer than ~40 minutes will execute in their own worker and extend the time that CI takes on all PRs and therefore must be split up. We had to do this with several existing FTR Configs like:
These config files are now broken up into many config files, many are per-app, but some apps needed to be broken down even further.
In order to run these tests locally contributors will need to pass the right config path based on the tests they want to run like: