unskip dashboard_screenshot tests by using new setScreenshotSize #46311

LeeDr · 2019-09-21T19:05:36Z

Summary

Added new browser.setScreenshotSize(width, height) method which figures the display scaling and browser boarders so it can setWindowSize to get the correct screenshot size.

Note that depending on the display scaling on the machine running the test, the session screenshot may not be the same size as the baseline screenshot but will proportional so that the comparePng method will deliver good results.

Should make screenshot tests work in both headless and headed Chrome. I haven't tried any of this on Firefox or IE yet.

Fixes: #40173
Fixes: #19471
(the second one could have been already fixed by some other change but since the test was skipped we didn't know.)

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately
We should not have to change 'notifications:lifetime:info': 10000 in config.js. I made this change because the notification for adding the first visualization to the dashboard was going away before the test could see it. It looks like it's going away before the default 5 seconds.

elasticmachine · 2019-09-21T22:07:24Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-22T04:02:46Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-23T15:13:04Z

@elasticmachine merge upstream

elasticmachine · 2019-09-23T17:06:13Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-23T17:50:37Z

This is why the second test that I unskipped is failing. It looks like it isn't transitioning into fullscreen mode, or hasn't completed that transition.
dashboard app using current data dashboard snapshots compare area chart snapshot

I don't know why I didn't see that locally.

This is the baseline image I saved locally;

UPDATE: I wasn't seeing this problem locally because I wasn't running the other dashboard tests. Once I ran at least one previous test the problem became clear. If we don't use one of the navigateToApp methods in the beginning of tests then our test will be impacted by the browser cache from the previous test. This impacts what colors appear in a new visualization.

The other problem with the top failing screenshot, is that it's NOT the screenshot used to do the diff. It's the screenshot captured automatically when the test fails. This test exits fullscreen mode before it does the expect on the screenshot. So when that expect fails (because of the colors in the chart) it was capturing the failure screenshot during the transition out of fullscreen. The method to exit fullscreen wasn't waiting for render complete, so we got the bad looking screenshot.

elasticmachine · 2019-09-23T21:23:58Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-24T00:08:49Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-24T03:56:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

dmlemeshko · 2019-09-24T11:14:43Z

@LeeDr , I did some testing on Macbook 15" with different display scaling:

Default (1680 x 1050): passed
1920 x 1200: passed
1440 x 900: passed
1280 x 800: both failed

        └- ✖ fail: "dashboard app using current data dashboard snapshots compare TSVB snapshot"
         │      Error: expected 0.039462 to be below 0.02

         └- ✖ fail: "dashboard app using current data dashboard snapshots compare area chart snapshot"
         │      Error: expected 0.035874 to be below 0.02

1024 x 640: both failed

✖ fail: "dashboard app using current data dashboard snapshots compare TSVB snapshot"
         │      Error: expected 0.099836 to be below 0.02
✖ fail: "dashboard app using current data dashboard snapshots compare area chart snapshot"
         │      Error: expected 0.174844 to be below 0.02

I think small resolutions may work better with an external display.

I want to point out that I have seen tests fail pretty often with the following error in ES logs (causing dashboard panel being empty):

info [r.suppressed] [Dzmitrys-MacBook-Pro.local] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=1000, index=.kibana, from=0}
   │      org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:314) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:273) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:424) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1091) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1200) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1174) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$7(SearchService.java:1055) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   │      	at java.lang.Thread.run(Thread.java:834) [?:?]

I was thinking it is caused by missing esArchiver unload, but was not able to prove it.

elasticmachine · 2019-09-24T20:42:10Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-24T21:21:59Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-24T21:51:37Z

@elasticmachine merge upstream

elasticmachine · 2019-09-24T22:08:22Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…rdSnapshots

jbudz · 2019-09-26T14:21:00Z

test/functional/apps/dashboard/dashboard_snapshots.js

      await dashboardPanelActions.clickExpandPanelToggle();
+      await PageObjects.common.sleep(5000);


are the sleeps still needed in addition to the window resize?

The sleeps were part of my debugging along with those additional screenshots. See the issue about the skip line but it appears to be a Kibana bug. But it never failed for me locally and always fails on Jenkins. Oh, that reminds me I was going to try it on my Ubuntu 18 machine...
Since someone should eventually debug and fix whatever the problem is, I thought it was OK to leave the sleeps and additional screenshots in this PR.

I unskipped the second test and ran it on my Ubuntu thinking it might reproduce the failure we see on Jenkins, but it didn't.

Correction: I did just reproduce it locally by running all the dashboard tests that come before this one. So something in a previous test is impacting this one. I'll work on figuring that out.

Update: I think I fixed the test. The problem was that the dashboard tests generally don't use the navigate methods which put the timestamp in the URL, which means that this test was impacted by the browser cache from the previous test and the charts don't get the same colors (which, of course, makes the screenshot comparison fail).
I did leave one commented-out 1/2 second sleep in this test. We'll see if we have any failures on opening the context menu without it.

LeeDr · 2019-09-26T15:50:51Z

@elasticmachine merge upstream

elasticmachine · 2019-09-26T17:07:38Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-26T19:19:00Z

The tests I unskipped passed, but this later dashboard test failed;

[00:10:30]                     │ debg Click Save Visualization button
[00:10:30]                     │ debg TestSubjects.click(confirmSaveSavedObjectButton)
[00:10:30]                     │ debg Find.clickByCssSelector('[data-test-subj="confirmSaveSavedObjectButton"]') with timeout=10000
[00:10:30]                     │ debg Find.findByCssSelector('[data-test-subj="confirmSaveSavedObjectButton"]') with timeout=10000
[00:10:30]                     │ debg isGlobalLoadingIndicatorVisible
[00:10:30]                     │ debg TestSubjects.exists(globalLoadingIndicator)
[00:10:30]                     │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:10:31]                     │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:10:31]                     │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:10:31]                     │ debg TestSubjects.exists(saveVisualizationSuccess)
[00:10:31]                     │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="saveVisualizationSuccess"]') with timeout=10000
[00:10:31]                     │ debg clickCancelOutOfEditMode
[00:10:31]                     │ debg TestSubjects.click(dashboardViewOnlyMode)
[00:10:31]                     │ debg Find.clickByCssSelector('[data-test-subj="dashboardViewOnlyMode"]') with timeout=10000
[00:10:31]                     │ debg Find.findByCssSelector('[data-test-subj="dashboardViewOnlyMode"]') with timeout=10000
[00:10:41]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="dashboardViewOnlyMode"])
[00:10:41]                     │      Wait timed out after 10056ms


✖ fail: "dashboard app using current data dashboard view edit mode shows lose changes warning and loses changes on confirmation when a new vis is added"
                   │      retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="dashboardViewOnlyMode"])

The screenshot is of the homepage;

At the moment, I have no idea how it got to the home page from this test. I'll try locally and run Jenkins again.

LeeDr · 2019-09-26T19:25:17Z

Jenkins retest

elasticmachine · 2019-09-26T19:41:20Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-26T19:50:04Z

Jenkins retest

LeeDr · 2019-09-26T19:55:37Z

Jenkins please test this

elasticmachine · 2019-09-26T21:07:27Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-26T23:54:44Z

I know we all hate the idea of this, but I had to re-arrange the order of the test suites so that the one I'm unskipping is at the end. For the details why, see #46752

elasticmachine · 2019-09-27T01:09:37Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

LeeDr · 2019-09-27T01:48:16Z

jenkins test this

elasticmachine · 2019-09-27T03:04:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

dmlemeshko

LGTM, I left a comment about commented sleep

dmlemeshko · 2019-09-27T12:17:17Z

test/functional/apps/dashboard/dashboard_snapshots.js

      await PageObjects.dashboard.clickFullScreenMode();
+      // await PageObjects.common.sleep(500);


@LeeDr if the test will pass on CI, can we remove it?

dmlemeshko · 2019-09-27T12:19:09Z

retest

elasticmachine · 2019-09-27T14:21:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

Hopefully this won't be an issue

elasticmachine · 2019-09-27T16:22:34Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

``` loadTestFile(require.resolve('./view_edit')); // Order of test suites *shouldn't* be important but there's a bug for the view_edit test above // elastic#46752 // The dashboard_snapshot test below requires the timestamped URL which breaks the view_edit test. // If we don't use the timestamp in the URL, the colors in the charts will be different. loadTestFile(require.resolve('./dashboard_snapshots')); ```

elasticmachine · 2019-09-27T18:12:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

use new setScreenshotSize

d383590

LeeDr added the test_ui_functional label Sep 21, 2019

Lee Drengenberg added 2 commits September 21, 2019 14:29

change tolerance from 1% to 2% (originally 5%)

b7125c7

retake baseline without notification dialog

851a4b4

close add and save notifications

be1111d

LeeDr added the test-matrix Use this label to ensure PRs are tested with matrix jobs label Sep 23, 2019

Merge branch 'master' into unskipDashboardSnapshots

3168644

LeeDr added 2 commits September 23, 2019 13:02

wait for render complete when going in to FullScreen mode

5bccbf4

add 5 second sleep to see if it ever goes to full screen mode

2522ab6

add debugging screenshots and sleeps

93fedae

LeeDr mentioned this pull request Sep 23, 2019

Failing test: Chrome UI Functional Tests.test/functional/apps/dashboard/dashboard_snapshots·js - dashboard app using current data dashboard snapshots compare area chart snapshot #40173

Closed

try reloading the Kibana index before this test

61d89c9

30 iterations with only the first test enabled

8ee4c58

revert multi-run changes

ae61a04

Merge branch 'master' into unskipDashboardSnapshots

4e2d3d2

LeeDr added 2 commits September 24, 2019 17:14

ran node scripts/build_renovate_config

c2722e8

Merge branch 'master' of github.com:elastic/kibana into unskipDashboa…

f0e8090

…rdSnapshots

LeeDr requested a review from liza-mae September 25, 2019 22:52

jbudz reviewed Sep 26, 2019

View reviewed changes

Merge branch 'master' into unskipDashboardSnapshots

38c2110

fixed and Unskipped 2nd test

69bf8e1

LeeDr mentioned this pull request Sep 26, 2019

Dashboard test fails to Home screen on dashboard app using current data dashboard view edit mode shows lose changes warning and loses changes on confirmation when a new vis is added #46752

Closed

re-arrange order of dashboard tests to pass :-(

8dc98f9

dmlemeshko approved these changes Sep 27, 2019

View reviewed changes

LeeDr removed the test-matrix Use this label to ensure PRs are tested with matrix jobs label Sep 27, 2019

removed commented-out sleep

220cc0b

Hopefully this won't be an issue

LeeDr removed the request for review from liza-mae September 27, 2019 18:49

LeeDr merged commit 5a92dec into elastic:master Sep 27, 2019

LeeDr mentioned this pull request Sep 30, 2019

[7.x] unskip dashboard_screenshot tests by using new setScreenshotSize (#46311) #46991

Merged

LeeDr deleted the unskipDashboardSnapshots branch August 20, 2020 17:54

		await dashboardPanelActions.clickExpandPanelToggle();
		await PageObjects.common.sleep(5000);

		await PageObjects.dashboard.clickFullScreenMode();
		// await PageObjects.common.sleep(500);

unskip dashboard_screenshot tests by using new setScreenshotSize #46311

unskip dashboard_screenshot tests by using new setScreenshotSize #46311

Conversation

LeeDr commented Sep 21, 2019 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Sep 21, 2019

💔 Build Failed

elasticmachine commented Sep 22, 2019

💔 Build Failed

LeeDr commented Sep 23, 2019

elasticmachine commented Sep 23, 2019

💔 Build Failed

LeeDr commented Sep 23, 2019 • edited Loading

elasticmachine commented Sep 23, 2019

💔 Build Failed

elasticmachine commented Sep 24, 2019

💔 Build Failed

elasticmachine commented Sep 24, 2019

💔 Build Failed

dmlemeshko commented Sep 24, 2019

elasticmachine commented Sep 24, 2019

💔 Build Failed

elasticmachine commented Sep 24, 2019

💔 Build Failed

LeeDr commented Sep 24, 2019

elasticmachine commented Sep 24, 2019

💔 Build Failed

jbudz Sep 26, 2019

Choose a reason for hiding this comment

LeeDr Sep 26, 2019

Choose a reason for hiding this comment

LeeDr Sep 26, 2019

Choose a reason for hiding this comment

LeeDr Sep 26, 2019

Choose a reason for hiding this comment

LeeDr Sep 26, 2019

Choose a reason for hiding this comment

LeeDr commented Sep 26, 2019

elasticmachine commented Sep 26, 2019

💚 Build Succeeded

LeeDr commented Sep 26, 2019

LeeDr commented Sep 26, 2019

elasticmachine commented Sep 26, 2019

💔 Build Failed

LeeDr commented Sep 26, 2019

LeeDr commented Sep 26, 2019

elasticmachine commented Sep 26, 2019

💔 Build Failed

LeeDr commented Sep 26, 2019

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

LeeDr commented Sep 27, 2019

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

dmlemeshko left a comment

Choose a reason for hiding this comment

dmlemeshko Sep 27, 2019

Choose a reason for hiding this comment

dmlemeshko commented Sep 27, 2019

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

LeeDr commented Sep 21, 2019 •

edited

Loading

LeeDr commented Sep 23, 2019 •

edited

Loading