Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix skipped test test/accessibility/apps/dashboard·ts - Dashboard create dashboard button #111233 #112872

Merged
merged 11 commits into from
Sep 28, 2021

Conversation

LeeDr
Copy link

@LeeDr LeeDr commented Sep 22, 2021

Summary

Fixes: #111233

Details in the issue ^. Although the before method should have added the sample data, the failure makes it look like it wasn't added. Although I initially tested adding a 2s sleep, I later changed it to a retry.waitFor.

Checklist

Delete any items that are not applicable to this PR.

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk Probability Severity Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space. Low High Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. High Low Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled. Medium High Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

@LeeDr
Copy link
Author

LeeDr commented Sep 22, 2021

@elasticmachine merge upstream

@tylersmalley
Copy link
Contributor

@elasticmachine merge upstream

@tylersmalley
Copy link
Contributor

@elasticmachine merge upstream

@tylersmalley
Copy link
Contributor

@elasticmachine merge upstream

@tylersmalley
Copy link
Contributor

This should do it - the problem was with CentOS, which is currently not in the pool.

@tylersmalley
Copy link
Contributor

I do want to confirm that we aren't adding sleeps to resolve this, however.

@LeeDr LeeDr changed the title test if 2s sleep fixes flakey test fix skipped test test/accessibility/apps/dashboard·ts - Dashboard create dashboard button #111233 Sep 24, 2021
@LeeDr
Copy link
Author

LeeDr commented Sep 24, 2021

Passed 100 iterations on flaky test runner

@LeeDr LeeDr added v7.16.0 v8.0.0 Project:Accessibility test_ui_functional auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes labels Sep 24, 2021
@LeeDr LeeDr marked this pull request as ready for review September 24, 2021 20:08
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-accessibility (Project:Accessibility)

@@ -17,15 +17,19 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) {
...functionalConfig.getAll(),

testFiles: [
require.resolve('./apps/discover'),
// these 5 tests all load addSampleDataSet('flights')
// only the last test does removeSampleDataSet('flights')
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: I started going down the path to create an index file with a before and after to load/unload the sample data for this group of tests but the last test kibana_overview appears to have tests for both states where there is and isn't data. I need to talk to @bhavyarm about refactoring these. But I'd like to get the test unskipped in any case.

@LeeDr LeeDr requested a review from wayneseymour September 24, 2021 20:11
@LeeDr
Copy link
Author

LeeDr commented Sep 28, 2021

@elasticmachine merge upstream

@LeeDr
Copy link
Author

LeeDr commented Sep 28, 2021

I do want to confirm that we aren't adding sleeps to resolve this, however.

No sleeps added. That was just to try to tell if it was a timing problem. I didn't really prove it one way or the other but added a retry in adding the sample data.

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@LeeDr LeeDr merged commit 200d035 into elastic:master Sep 28, 2021
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Sep 28, 2021
…ate dashboard button elastic#111233 (elastic#112872)

* test if 2s sleep fixes flakey test

* switch from sleep to retry.waitFor

* remove .only, add comments to config file about sample data add/remove

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Sep 28, 2021
…ate dashboard button #111233 (#112872) (#113317)

* test if 2s sleep fixes flakey test

* switch from sleep to retry.waitFor

* remove .only, add comments to config file about sample data add/remove

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Lee Drengenberg <lee.drengenberg@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes test_ui_functional v7.16.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failing test: Accessibility Tests.test/accessibility/apps/dashboard·ts - Dashboard create dashboard button
5 participants