Weird test data for saved objects

[jportner]

🗒 Note

This is currently labeled with Team:QA, but I'm envisioning this as a shared responsibility across Kibana teams with QA as a "primary owner".

Overview

Kibana currently provides three sample datasets (Ecommerce, Flights, and Logs) -- whenever you add one of these datasets from the homepage, it actually loads two groups of data:

1. Sample documents in the ES sample data index
2. Sample saved objects (data view, visualizations, dashboard, etc.) that make use of this sample data index

Kibana's saved objects are very flexible, they can be strongly linked (by references) or weakly linked (by markdown and/or handlebars), and some saved objects are also "embeddables" that can be rendered in other apps. This is further complicated by the fact that starting in 8.0, saved object IDs can change (both on upgrade and on import/copy), and legacy URL aliases can get created in certain conditions.

What to do

We have had trouble testing all of the complex ways that saved objects can be loaded and linked to in Kibana. It would be helpful to have more test data for saved objects that allows engineers to test these different scenarios. Here's what I'm envisioning:

1. Add a 4th "weird" test dataset: This can reuse sample documents from one of the other three datasets, but it would provide new/varied sample saved objects for engineers to test with.
2. Change the Load Sample Data screen: If you are running Kibana from source, and/or if you enable a flag in Advanced Settings, we show a 4th option to load this weird test dataset.
   -> This will be much more visible and easily usable for engineers to smoke-test their saved objects!
3. Add an abstraction in the functional test suite: This will allow consumers to easily leverage this weird test dataset in their end-to-end tests.

Using dashboards as an example:

• A dashboard with tons of different visualizations (100+)
• Four permutations of dashboards with visualizations:
  [Dashboard saved object | Dashboard snapshot URL] + [Visualizations "by-reference" | Visualizations "by-value"]
• Different dashboards with “weak links” to each other
• Dashboards in multiple spaces which may or may not have been migrated from an old version of Kibana, and were imported/copied from one space to another with different options:
  [Existed before upgrading to 8.0 | Do not already exist] + [Create new copies | Check for existing objects w/o overwriting | Check for existing objects and automatically overwrite]
• Data view without a time field

Scope

Initially, I think it makes sense to start small and put this together only for Kibana team plugins.
As a follow-up, it would be good to include solutions as well (Observability and Security Solution -- Enterprise Search doesn't use saved objects right now).

Other thoughts

• We want to move towards using the kbn-archiver package to load saved objects for functional testing. However, some of the saved objects that we might want to work with may not be importable/exportable. We'll need to figure out a solution to this if we want to stick with kbn-archiver -- otherwise we may need to resort to es-archiver instead.
• Maintenance of this new weird test dataset is a concern here, and historically sample data has not been well-maintained. We will need each contributing team to adopt a strategy to maintain this data on a periodic basis.
• If we have a single shared dataset that many teams contribute to, there's always a chance that one team's changes (adding or updating a saved object) could break something else unexpectedly. We should aim to have some basic tests in place (screenshot tests??) to make sure that stuff doesn't break unexpectedly.
• Sample documents are loaded relative to the current time -- this means it would be difficult to test with. We should consider loading sample documents for this weird test dataset with an absolute time instead.
• It would be nice to include "diagnostic" dashboards (for the .kibana and/or .kibana_task_manager indices) too.

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[jportner]

CC @flash1293 @tylersmalley

[bhavyarm]

@jportner what does "Different dashboards with “weak links” to each other" mean? Sorry for asking again.
Oh also drilldowns? Thanks!

[jportner]

@jportner what does "Different dashboards with “weak links” to each other" mean? Sorry for asking again. Oh also drilldowns? Thanks!

Good question! This was inspired by #123550. "Weak links" is the term we are using to describe any free-text that links to a saved object. For example, a dashboard 123 (accessible on http://localhost:5601/app/dashboards#/view/123) may have a Markdown visualization that contains a relative hyperlink to a different dashboard at dashboards#/view/456.

Elastic actually ships some visualizations that use such "weak links" today, apparently Beats dashboards makes use of this.

[LeeDr]

It might also be helpful to find a good way to list/describe what we have for existing data sets. For example, @bhavyarm just added a new test with dashboards full of visualizations by-reference and by-value with both importing into multiple spaces, and copy-to-space tests.