Generate doc screenshots from tests

[rashidkpc]

We need to define the steps the user would take to to get to the screenshots in the docs, then write tests to get the screen to this point and take a screenshot. This will ensure our docs are correct, because if they aren't, the tests will break

@palecur please work with @LeeDr to get up to speed on the integration testing and screenshot system

[palecur]

@cjcenizal has this down, the kibana-ci pages have what I need now, closing this thereby

[LeeDr]

We're now saving 86 screenshots through the tests;

baseline/Console-default-request.png
baseline/Console-help-collapsed.png
baseline/Console-help-expanded.png
baseline/Dashboard-add-visualizations.png
baseline/Dashboard-has-visualizations.png
baseline/Dashboard-load-saved.png
baseline/Dashboard-no-visualizations.png
baseline/Dashboard-set-timepicker.png
baseline/Dashboard-visualization-sizes.png
baseline/Discover-copy-to-clipboard-toast.png
baseline/Discover-field-data.png
baseline/Discover-load-query.png
baseline/Discover-no-results.png
baseline/Discover-save-query-toast.png
baseline/Discover-share-link.png
baseline/Discover-shorten-url-button.png
baseline/Discover-sidebar-collapsed.png
baseline/Discover-sidebar-expanded.png
baseline/Discover-sort-down.png
baseline/Discover-syntax-error-toast.png
baseline/Settings-advanced-tab.png
baseline/Settings-indices-column-name-sort-ascending.png
baseline/Settings-indices-column-name-sort-descending.png
baseline/Settings-indices-column-type-sort-ascending.png
baseline/Settings-indices-column-type-sort-descending.png
baseline/Settings-indices-confirm-remove-index-pattern.png
baseline/Settings-indices-enable-creation.png
baseline/Settings-indices-new-index-pattern.png
baseline/Settings-indices-paged.png
baseline/Settings-indices-result-popularity-saved.png
baseline/Settings-indices-result-popularity-updated.png
baseline/Settings-initial-state.png
baseline/Settings-set-timezone.png
baseline/Status.png
baseline/Visualize-area-chart-save-toast.png
baseline/Visualize-area-chart.png
baseline/Visualize-chart-types.png
baseline/Visualize-data-table.png
baseline/Visualize-line-chart.png
baseline/Visualize-metric-chart.png
baseline/Visualize-pie-chart.png
baseline/Visualize-site-map.png
baseline/Visualize-vertical-bar-chart.png
session/Console-default-request.png
session/Console-help-collapsed.png
session/Console-help-expanded.png
session/Dashboard-add-visualizations.png
session/Dashboard-has-visualizations.png
session/Dashboard-load-saved.png
session/Dashboard-no-visualizations.png
session/Dashboard-set-timepicker.png
session/Dashboard-visualization-sizes.png
session/Discover-copy-to-clipboard-toast.png
session/Discover-field-data.png
session/Discover-load-query.png
session/Discover-no-results.png
session/Discover-save-query-toast.png
session/Discover-share-link.png
session/Discover-shorten-url-button.png
session/Discover-sidebar-collapsed.png
session/Discover-sidebar-expanded.png
session/Discover-sort-down.png
session/Discover-syntax-error-toast.png
session/Settings-advanced-tab.png
session/Settings-indices-column-name-sort-ascending.png
session/Settings-indices-column-name-sort-descending.png
session/Settings-indices-column-type-sort-ascending.png
session/Settings-indices-column-type-sort-descending.png
session/Settings-indices-confirm-remove-index-pattern.png
session/Settings-indices-enable-creation.png
session/Settings-indices-new-index-pattern.png
session/Settings-indices-paged.png
session/Settings-indices-result-popularity-saved.png
session/Settings-indices-result-popularity-updated.png
session/Settings-initial-state.png
session/Settings-set-timezone.png
session/Status.png
session/Visualize-area-chart-save-toast.png
session/Visualize-area-chart.png
session/Visualize-chart-types.png
session/Visualize-data-table.png
session/Visualize-line-chart.png
session/Visualize-metric-chart.png
session/Visualize-pie-chart.png
session/Visualize-site-map.png
session/Visualize-vertical-bar-chart.png

[cjcenizal]

@palecur I'm reopening and assigning this to myself. I'm working on setting up an isolated series of functional tests which will take screenshots to help us guard against visual regressions. I also got some feedback from @kurtado that indicated it will be useful to have a series of tests that take screenshots for the training (which was the original intention behind this ticket). I'll close this once those two functional test batteries are done.

[LeeDr]

@cjcenizal I'm thinking we should just close this? We did take a bunch of screenshots but they were never used in any docs. I think we've pulled them out at this point, or at least agreed that we would.

[cjcenizal]

@LeeDr Are you referring to the screenshots we took for visual diffing? If so, then I'd say this issue is a separate topic, so I'd consider it apart from those screenshots.

I do think this issue could be reframed as "Automate documentation screenshots", because that process is currently manual and prone to human error. I'm not sure we need to couple that process to our tests. If you want to close this and create a new issue to capture that requirement, I think that would make sense. What do you think?

[LeeDr]

I'm not referring to the visual diffing screenshots. This issue is about capturing screenshots during the tests while on various pages. I count 44 of them on master. As far as I know nobody has ever used them.

Lee@LeeD-ASUS MINGW64 /c/git/kibana (master)
$ grep -r "screenshots.take(" test/*
test/functional/apps/console/_console.js:      screenshots.take('Console-help-expanded');
test/functional/apps/console/_console.js:        screenshots.take('Console-help-collapsed');
test/functional/apps/console/_console.js:        screenshots.take('Console-default-request');
test/functional/apps/dashboard/_dashboard.js:      await screenshots.take('Dashboard-no-visualizations');
test/functional/apps/dashboard/_dashboard.js:      await screenshots.take('Dashboard-add-visualizations');
test/functional/apps/dashboard/_dashboard.js:      await screenshots.take('Dashboard-load-saved');

...

[cjcenizal]

@LeeDr Oh, I see what you mean. Yes, personally I don't think we need to capture those screenshots. In fact, I created #11350 which calls for their removal.

[palecur]

Historical note: These were originally meant to keep the images in the Kibana documentation up to date, but I suppose there's another approach to that nowadays.

[cjcenizal]

I did a cursory search through the current /docs directory and couldn't find references to the screenshots we're saving, so it's possible we're no longer using them for their original purpose (or at all).

If we decide to re-implement the automation of documentation screenshots (which I think is a great idea), I think we should create a doc_screenshots.js file, which exports the names of the files we want to save:

export const DOC_SCREENSHOT_CONSOLE_HELP_EXPANDED = 'console-help-expanded';
export const DOC_SCREENSHOT_CONSOLE_HELP_COLLAPSED = 'console-help-collapsed';
// etc...

Then we can reference these names from within the test:

screenshots.take(DOC_SCREENSHOT_CONSOLE_HELP_EXPANDED);

Benefits:

1. The role of these screenshots will be clear so we won't get confused about why they're there.
2. We can scan the doc_screenshots.js file to see a list of all screenshots we're generation, so we can remove ones that are no longer used in the docs.

We should probably also set up an independent set of "tests" to uncouple this process from our functional tests.