Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky vega screenshot test #33187

Closed
lukeelmers opened this issue Mar 13, 2019 · 4 comments · Fixed by #72238
Closed

Fix flaky vega screenshot test #33187

lukeelmers opened this issue Mar 13, 2019 · 4 comments · Fixed by #72238
Assignees
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Vega Vega visualizations Feature:Visualizations Generic visualization features (in case no more specific feature label is available) skipped-test Team:Visualizations Visualization editors, elastic-charts and infrastructure test_ui_functional test

Comments

@lukeelmers
Copy link
Member

#32749 introduced a screenshot-based functional test for validating the output of vega visualizations. this test was subsequently skipped in #33074 due to flakyness.

most other functional tests for visualizations rely on tabular data from the inspector to assert that the right data are being displayed, but since vega doesn't have inspector, this isn't an option.

the other place we are are using screenshot tests is in the dashboard, where we have the option of using full screen mode to maintain focus on the visualization without worrying about things like the nav evolving over time. unfortunately, full screen isn't an option within the visualize editor.

what we've tried:

  1. (current approach) inject some CSS to the DOM, hiding unnecessary elements, and then take the screenshot. this passes consistently in Chrome on OSX, but when run in CI will usually fail.
  2. inspect the DOM, find the vega canvas element, call canvas.toDataUrl() and compare the base64-encoded output. in initial testing, this proved problematic as it seemed rendering the vis with the same filters didn't always reproduce the exact same encoded output... though might be worth revisiting to test further.
  3. messing with the variance threshold for the screenshot comparison. this is a challenge because it needs to be sensitive enough to detect small differences in line charts, and raising the threshold too high will result in false positives.
@lukeelmers lukeelmers added test Feature:Visualizations Generic visualization features (in case no more specific feature label is available) Feature:Vega Vega visualizations test_ui_functional failed-test A test failure on a tracked branch, potentially flaky-test skipped-test labels Mar 13, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage

@lukeelmers lukeelmers added the Team:Visualizations Visualization editors, elastic-charts and infrastructure label Mar 13, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app

@flash1293
Copy link
Contributor

@lukeelmers This is probably not super present (given it's more than a year old), but I still want to ask whether you tried using the svg renderer for vega ("config": { "kibana": {"renderer": "svg"} } as part of the spec). Then we can validate whether the svg elements are set up correctly (with some allowance for sub pixel differences as well).

If you didn't try that, I'm happy to pick up the task.

@flash1293 flash1293 self-assigned this Jul 14, 2020
@lukeelmers
Copy link
Member Author

I still want to ask whether you tried using the svg renderer for vega ("config": { "kibana": {"renderer": "svg"} } as part of the spec

Nope, definitely did not consider this approach, so feel free to pick it up -- Thanks @flash1293!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Vega Vega visualizations Feature:Visualizations Generic visualization features (in case no more specific feature label is available) skipped-test Team:Visualizations Visualization editors, elastic-charts and infrastructure test_ui_functional test
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants