Fix a few dashboard flaky tests #22901

stacey-gammon · 2018-09-10T20:37:26Z

After discover the "slow 3g" option in Chrome, I was able to reproduce many flaky dashboard issues.

clickNewDashboard checked for an add dashboard button, if it didn't exist, it tried until failure to find the button on the prompt. Problem is sometimes it was slow to render the page, so dashboards existed, and the add dashboard button just didn't render yet. But once it failed to find it, it assumed there were no dashboards and only waited to find the "add your first dashboard" prompt button.
similarly, gotoDashboardLandingPage first checked for the existence of an item to determine where it was, but if that came back incorrectly (because the page was slow to load), it would fail. Wrapping both checks in a retry should fix this.
The second click save button happened too quickly, this waits for the button to be enabled before clicking. Clicking the button when disabled doesn't throw an error. Rather than fix for this specific test, I pushed the check for a disabled button into the find service so any other clicks on an element will wait for the button to be enabled, or will throw an error as unsuccessful.
Added in a check to wait for the EuiTableLoading indicator to be removed after searching for a dashboard on the listing page. If it's in the process of loading, some clicks can miss their elements.

Fixes #22409 ... probably many more! 🎉

stacey-gammon · 2018-09-10T20:39:34Z

test/functional/apps/visualize/_linked_saved_searches.js

@@ -24,79 +24,76 @@ export default function ({ getService, getPageObjects }) {
 const retry = getService('retry');
 const PageObjects = getPageObjects(['common', 'discover', 'visualize', 'header']);

- describe('visualize app', function describeIndexTests() {


just removed the unnecessary wrapper

elasticmachine · 2018-09-10T22:10:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-09-11T00:55:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-09-11T13:45:34Z

retest

elasticmachine · 2018-09-11T15:14:42Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-09-11T15:53:51Z

retest

elasticmachine · 2018-09-11T17:12:05Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-09-11T17:31:11Z

Failed on a flaky test

UI Functional Tests.test/functional/apps/dashboard/_dashboard_state·js.dashboard app using legacy data dashboard state Overriding colors on an area chart is preserved

jenkins, test this

elasticmachine · 2018-09-11T18:50:53Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-09-11T18:57:14Z

Overriding colors on an area chart is preserved failure again. This one I guess needs to be addressed now, something in this PR may be causing it to be hit more frequently.

Generalize the fix and include some other flakiness encountered with slow network connection dont wrap in a retry or stale element checks will get caught dont click disabled go button Skip test hitting legit input control bug

elasticmachine · 2018-09-13T12:49:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-09-13T13:13:36Z

Failed on UI Functional Tests.test/functional/apps/visualize/_data_table·js.visualize app data table should show correct data for a data table with top hits

jenkins, test this

LeeDr · 2018-09-13T13:18:46Z

This last failure was;
"FAILURE test/functional/apps/visualize/_data_table·js visualize app data table should show correct data for a data table with top hits"

That test has only failed 2 other times in the last 30 days. Both of those were on Sept 11th on @ppisljar's PR #22756 (which is now merged) but I don't know if his PR had anything to do with that test. I don't see any changes to tests in that PR.

If this PR is adding the check to see if something is disabled then this failure might be related (this is from this PR's last failing build);

           │ debg TestSubjects.find(globalLoadingIndicator-hidden)
           │ debg findByCssSelector [data-test-subj~="globalLoadingIndicator-hidden"]
           │ debg clickByCssSelector(button[data-test-subj="toggleEditor"])
           │ debg findByCssSelector button[data-test-subj="toggleEditor"]
           │ debg clickWhenNotDisabled
           │ debg clickByCssSelector(
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .agg-select
           │            )
           │ debg findByCssSelector 
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .agg-select
           │            
           │ debg clickWhenNotDisabled
           │ debg findByCssSelector 
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .agg-select
           │             input.ui-select-search
           │ debg ... sleep(500) start
           │ debg ... sleep(500) end
           │ debg clickByCssSelector(
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .field-select
           │            )
           │ debg findByCssSelector 
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .field-select
           │            
           │ debg clickWhenNotDisabled
           │ debg findByCssSelector 
           │              [group-name="metrics"]
           │              vis-editor-agg-params:not(.ng-hide)
           │              
           │              .field-select
           │             input.ui-select-search
           │ debg ... sleep(500) start
           │ debg ... sleep(500) end
           │ debg TestSubjects.click(visualizeEditorRenderButton)
           │ debg clickByCssSelector([data-test-subj~="visualizeEditorRenderButton"])
           │ debg findByCssSelector [data-test-subj~="visualizeEditorRenderButton"]
           │ debg clickWhenNotDisabled
           │ debg Element is disabled, try again
           │ debg Element is disabled, try again
           │ debg Element is disabled, try again
           │ debg Element is disabled, try again

It doesn't look like we captured a screenshot or page source for some reason on this failure so it's hard to know why the visualizeEditorRenderButton was not enabled.

stacey-gammon · 2018-09-13T13:29:09Z

it might be a legit bug. I ran into a situation last week where the play button became disabled and it wasn't becoming enabled. I couldn't repro through, once I refreshed the page.

I won't submit until these issues are worked out, but this is partly why I'm concerned that swapping out an entire test backend for all tests is going to be a huge issue. Slight timing changes can expose legit bugs and more flakiness.

Look at this PR: #22787

When I ran the _dashboard_filtering test suite multiple times in a row, a test farther down the line failed for an unrelated error. I would like to dig into that more to get to the root of the problem, but it seems very easy to break a bunch of tests by just changing the timing of things up a bit. That PR also hit quite a few unrelated flaky tests, though at least stopped failing consistently when I went back to only a single test run.

Overriding colors on an area chart is preserved also seems to be getting hit more frequently in this PR. I might explore breaking up some of the improvements in this PR to get some of them checked in.

elasticmachine · 2018-09-13T14:32:35Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon force-pushed the 2018-09-06-fix-flaky-tests branch from b3465cc to 3c8623c Compare September 10, 2018 20:38

stacey-gammon commented Sep 10, 2018

View reviewed changes

stacey-gammon force-pushed the 2018-09-06-fix-flaky-tests branch from 5776953 to 28a2a5e Compare September 10, 2018 23:25

stacey-gammon requested review from LeeDr and liza-mae September 11, 2018 15:54

stacey-gammon mentioned this pull request Sep 11, 2018

The future of our Functional Test Runner #22947

Closed

stacey-gammon added 3 commits September 13, 2018 07:29

Make sure the click is only when the button is enabled.

c8bd6c8

Generalize the fix and include some other flakiness encountered with slow network connection dont wrap in a retry or stale element checks will get caught dont click disabled go button Skip test hitting legit input control bug

couple more places clicking go did nothing.

13c20a5

Add a waitForRenderComplete after add and dashboard save

d07b314

stacey-gammon force-pushed the 2018-09-06-fix-flaky-tests branch from 28a2a5e to d07b314 Compare September 13, 2018 11:31

stacey-gammon mentioned this pull request Sep 20, 2018

Use concrete assertions, not exists #23271

Merged

stacey-gammon closed this Oct 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix a few dashboard flaky tests #22901

Fix a few dashboard flaky tests #22901

stacey-gammon commented Sep 10, 2018

stacey-gammon Sep 10, 2018

elasticmachine commented Sep 10, 2018

elasticmachine commented Sep 11, 2018

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 13, 2018

stacey-gammon commented Sep 13, 2018

LeeDr commented Sep 13, 2018

stacey-gammon commented Sep 13, 2018

elasticmachine commented Sep 13, 2018

Fix a few dashboard flaky tests #22901

Fix a few dashboard flaky tests #22901

Conversation

stacey-gammon commented Sep 10, 2018

stacey-gammon Sep 10, 2018

Choose a reason for hiding this comment

elasticmachine commented Sep 10, 2018

💔 Build Failed

elasticmachine commented Sep 11, 2018

💚 Build Succeeded

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

💚 Build Succeeded

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

💔 Build Failed

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 11, 2018

💔 Build Failed

stacey-gammon commented Sep 11, 2018

elasticmachine commented Sep 13, 2018

💔 Build Failed

stacey-gammon commented Sep 13, 2018

LeeDr commented Sep 13, 2018

stacey-gammon commented Sep 13, 2018

elasticmachine commented Sep 13, 2018

💔 Build Failed