Scale tick tests are flaky

[markov00]

UPDATE LeeDr - please list the completed name of at least one test so we can find the issues;
"visualize app line charts switch between Y axis scale types should show ticks on selecting log scale"

These tests are currently flaky because the window sizes between local test environment and the CI environment differs.

The tests are meant to checks for correct displaying of vertical axis tick labels. The placement and their values depends on the height of the chart, and in visualize editor the height is determined by the height of the window.

I've two different situation on local machine (osx) and on CI:

CI (report a screenshot size of 1280 × 695) with 20 ticks ( 2,3,4,6,8 and their 10x multiple)
local (report a screenshot size of 2560 × 1372 2x dpi because on retina) 19 ticks (2,3,5,7,10 and their 10x multiples minus 10.000)
So on local machine is taken ad 2x density pixel. the different is on height.
The default configured height in leadfoot is 1280 x 800 and

CI have a top chrome space of 105 pixels
local have a top chrome space of 228 pixel at 2x, that means 114 pixels
A difference of 9 pixels in height between the two environments.

CI screenshot

Local screenshot

[markov00]

@silne30 Do you know if it’s possible to set the window innerHeight instead of the outerHeight when setting the windowsSize with leadfoot?

[markov00]

@liza-mae What does "done" means on https://github.com/elastic/kibana/projects/6#card-11860491 ?when the test is skipped or when the flaky test became a stable test? I've already skipped the test on this PR #21641 but I want to keep open this issue so I can find a better solution later

[liza-mae]

@markov00 good point, 'Done' should mean the flaky test became stable test, for skipped test we can leave in 'ToDo' or we can create another column 'Skipped' - a test can stay in that state until it is re-enabled, how does that sound?

[markov00]

@liza-mae I think adding a column skipped can help us to identify the status of an issue. I can move this issue to the skipped, and than move to done when finally the test became stable.

[liza-mae]

@markov00 - Thanks! I agree, I have added a skipped column.

[cuff-links]

@markov00 There is no indication of a way to specify but Leadfoot has been known to welcome updates and PRs. https://theintern.io/leadfoot/module-leadfoot_Command.html#setWindowSize
Perhaps a PR to where you can set a flag for innerHeight and outerHeight would be good to do.

[chrisdavies]

In reporting, instead of setting the window height, we set the containing element height. That way, no matter the window size, we get a consistent height in reports. It's hacky, but it'd make these tests predictable.

[rayafratkina]

Removing the version labels: we should not be assigning version until the PR is merged.

[stacey-gammon]

@markov00 - you are assigned, are you still working on this, or should this be closed? I think those tests are still being skipped...

[markov00]

The tests are skipped. There isn't a valid solution to that problem, neither leadfoot or webdriver have a way to reliably set the page height as described on that issue. I've talked to @dmlemeshko today, he will check if it's possible, at least in chrome.

[markov00]

hey @dmlemeshko any news on this topic?