Dashboards automate backwards compatibility tests

[kavilla]

Backward Compatibility (BWC) test is used to verify the compatibility of the current developed software with their older versions. Specifically, for OSD which is forked from Kibana v7.10.2, BWC test means verification to see whether customers can upgrade from old Kibana versions to OSD and from old OSD versions to the current one. Currently BWC is done manually for each release and it takes about 12-15 hrs. With more incoming features and changes in each release, it is demanding for us to automate the BWC test to guarantee that there is no breaking changes between versions.

Requirement

• Add BWC tests
• Add a script in OpenSearch-build repo to call BWC test for OpenSearch Dashboards and plugins during each release
• Add a Github action to trigger BWC test for PR pull and merge

Describe the solution you'd like

Data stored from previous version and it automates the testing of compatibility. Below is the details

Test Framework: Cypress

For the test framework, we decide to use Cypress. Below are some considerations:

• Re-use the code from summer project ( functional-test and functional-library ) and help to complete the library for future usage. It is much easier for OSD core to pick up. With a complete library and well-defined page components, it is also easier for customers to apply and write their own functional tests.
• Easier for plugin owners to pick up: Plugins already use Cypress as their test framework. It is easier for them to pick up or apply the library if needed.
• OpenSearch-build team is working on supporting Cypress in their pipelines.

Test Location:

We will first create the test in Dashboards own Repo Repo here. However, it can't be run directly in Dashboards because we will not add cypress as dev-dependency for now due to version conflicts (cypress v6.9.1 requires node v12+). After node version bumped to v12+, we will add cypress as dev-dependency.

• Pros: Plugins don’t need to maintain individual version of Cypress. Here is the open issue.
• Cons: Cypress test is full dependent on OSD running.

Test data

We still need to manually create and save data for each old versions. Below is how we create test data.

Open docker and run OpenSearch and Dashboards. Navigate to localhost:5601 and do the following

• Add data and verify
  • open home page
  • call add_sample_data fun to add all sample data. verify sample data are added
  • verify flight has maps and notice maps working
• Add Timelion/Timeline visualization and verify
  • open visualize page
  • create visualization and click Timelion/Timeline
  • save with name test-timelion/test-timeline and description
  • verify Timelion/Timeline visualization is saved
  • add the expression kibana=false (issue here ([BUG] Timeline function changed to opensearchDashboards and filters not applying if set to true OpenSearch-Dashboards#820)) and verify if we see any errors
• Add filter & query and verify
  • open Dashboard page and click [Logs] Web Traffic
  • add filter: Field: machine.os Operator: is Value: osx Custom label: osx filter
  • add query: resp=200 and save it with name test-query and description
  • verify the filter and query work properly
• Update default page and verify
  • open overview page
  • click make this my landing page and verify it is now the landing page
• Update settings and verify
  • open stack management page
  • click Advanced Settings
  • click to set Dark Mode
  • set Timelion/Timeline:default_columns: 4
  • set Timelion/Timeline:max_buckets: 4
  • verify all the updated settings

Saved the test data to local from docker. For example:

docker cp opensearch-node:/usr/share/opensearch/data/. ./data

Test cases

In Dashboards: navigate to localhost:5601 and do the following check with cypress tests

• Verify data are added and maps work well
  • open home page
  • verify all the sample data are added
  • verify flight has maps and notice maps working ??
• Verify Timelion visualization
  • open visualize page
  • verify there is a saved timeline visualization with name test-timelion and description
  • add the expression opensearchDashboards=false and verify if we see any errors
  • add the expression kibana=false and verify if we see any errors
• Verify filter & query
  • open Dashboard page and click [Logs] Web Traffic
  • verify filter exist and working: Field: machine.os Operator: is Value: osx Custom label: osx filter
  • verify query exist and working: resp=200 and save it with name test-query and description
• Verify default page and verify
  • open overview page
  • verify overview page is the default landing page
• Update settings and verify
  • open stack management page
  • click Advanced Settings
  • verify it is set to Dark Mode
  • verify Timeline default_columns equal to 4
  • verify Timeline max_buckets equal to 4

[kavilla]

Re-opened to add the CI/CD