Plugin E2E: Enable assertion on panel data #688

sunker · 2024-01-23T09:49:06Z

What this PR does / why we need it:

Plugin E2E currently provides two ways to assert that a data source query works as expected:

using the toBeOK matcher to check that the data query response was ok
e.g await expect(explorePage.runQuery()).toBeOK();
using the hasPanelError matcher to checks weather the panel has an error icon

As @yesoreyeram pointed out, we need a way to check what data is being displayed in the panel too. That way, one can assert that the entire data flow of a plugin (data source plugin in particular) works as expected.

Asserting on the data that is being displayed in a panel is tricky as the markup is a moving target. For that reason, it's important that we somehow incapsulate panel data extraction within plugin-e2e so we have a chance to fix the parsing whenever the panel markup changes. I was initially considering writing a plugin-e2e custom matcher to cater for this. However, that may lead to a scenario where we need to provide a whole bunch of different matchers to allow users to find assert on single value, multiple values, substring searches etc. It would be better if we could find a way to use the built-in Playwright matchers.

In this PR, the Panel model exposes two methods - getFieldNames and getData. getFieldNames returns a locator that resolves to element(s) that point to data frame field name, and getData returns a locator that resolves to element(s) that point to data frame values. Since they both return a Playwright locator, it can be used together with any of built-in playwright matchers. Currently, this only works for the table panel. That is likely the only panel you'd want to use in an E2E test anyway, but if another panel is being used one can always toggle View table (in panel edit), and then assertion should work.

e.g

test('table panel data assertions', async ({ page, selectors, grafanaVersion, request, readProvision }) => {
    const dashboard = await readProvision<Dashboard>({ filePath: 'dashboards/redshift.json' });
    const panelEditPage = new PanelEditPage({ page, selectors, grafanaVersion, request }, { dashboard, id: '3' });
    await panelEditPage.goto();
    await panelEditPage.setVisualization('Table');
    await expect(panelEditPage.panel.getData()).toContainText(['staging']);
    await expect(panelEditPage.panel.getFieldNames()).toContainText(['time', 'temperature']);
});

You'll find more examples of how data assertions can be done in provisioned dashboard, explore page etc in the dataAssertion.spec.ts file.

📦 Published PR as canary version: Canary Versions

✨ Test out this PR locally via:

npm install @grafana/plugin-e2e@0.13.0-canary.688.5df6275.0
# or 
yarn add @grafana/plugin-e2e@0.13.0-canary.688.5df6275.0

github-actions · 2024-01-23T09:49:28Z

Hello! 👋 This repository uses Auto for releasing packages using PR labels.

✨ This PR can be merged and will trigger a new minor release.
NOTE: When merging a PR with the release label please avoid merging another PR. For further information see here.

sunker · 2024-01-24T07:46:11Z

packages/plugin-e2e/src/models/panels/visualizations.ts

+  }
+
+  getLabels(): Locator {
+    return this.panel.locator('[role="columnheader"]');


Of course we'd need to add proper e2e selectors to all visualizations in Grafana an use them instead of week magic string like this (the same goes for all following extractors).

yesoreyeram

Feels like a good start. We also need to consider multiple cases and make it more generic. In general, instead of implementing individual methods for panels such as getLabels/getValues, I would like to see getData kind of abstraction which should return array of data frames kind of data used by the viz to render the data. We could use toggle table view and extract data to make it more consistent for the viz.

If I am datasource developer, I will least bother about how different visualisation render my data. Only care about if the visualization getting correct/consistent data from the query.

packages/plugin-e2e/src/models/panels/visualizations.ts

yesoreyeram · 2024-01-25T09:38:36Z

packages/plugin-e2e/src/models/panels/visualizations.ts

+    return this.panel.locator('[role="columnheader"]');
+  }
+
+  getValues(): Locator {


Did we consider the scenario where a data query respond with multi frame?

hmm that's the wide format right? doesn't the table panel give you the option to change series from a dropdown then? maybe plugin-e2e could provide apis for changing that.

that's the multi frame format where you see the dropdown with frame names. This dashboard may be handy (you can change scenario and play around).

yesoreyeram · 2024-01-25T09:40:55Z

packages/plugin-e2e/src/models/panels/visualizations.ts

+  }
+};
+
+export class VisualizationDataExtractor {


In general, when we are are edit panel view it would be easy to extract the data from any visualization by toggle to "table view". Instead of reading data for all the viz in unique way, this will simplify most of the work.

Yeah you may be right. I think I mostly wanted to support different visualizations because it would partly enable E2E testing that a provisioned databoard is working as expect.

const dashboard = await readProvision<Dashboard>({ filePath: 'dashboards/redshift.json' }); const dashboardPage = new DashboardPage({ page, selectors, grafanaVersion, request }, { uid: dashboard.uid }); await dashboardPage.goto(); // this is a Gauge panel const panel = await dashboardPage.getPanelByTitle('Redshift'); expect(await panel.getValues()). toHaveValue('test')

But ofc this would work also by switching to table view.

In enterprise datasources, we mostly rely on explore view for simplicity. That workflow directly opens the explore URL with the query passed in the param and then assert the table results. Then optionally you can change the values in the editor to ensure frontend components works as expected. But unfortunately, that workflow doesn't cover cases such as variable interpolation.

sunker · 2024-01-25T11:40:04Z

Thanks for feedback @yesoreyeram!

Yes I like the idea of toggling to table view for all panels. Will make it way less complicated.

I would like to see getData kind of abstraction which should return array of data frames kind of data used by the viz to render the data.

How would we be able to get the data frames passed to a panel from within an E2E test? Checking the Query inspector panel data would be slow and brittle.

I'm not sure you would want to assert on content of the data frames in an E2E test. Even though the format is to be considered stable, it's still moving target so you'd need to change your tests whenever a change is introduced. I think we're better of looking at what's being rendered in the table panel. If the rendering of table data is changed, we would catch that in an E2E test in grafana core and provide a fix in the plugin-e2e parser.

What about if we skip the whole vizualization concept, and instead expose two methods from the panel model - getData and getFieldNames (or getColumnNames). These methods would first toggle to Table view and then return a Playwright locator which will give plugin authors access to the variety of Playwright matchers that already exist (there's a lot of them). Then if there's a need to assert on the shape of the data frames, they can use the json in the response returned by for example panelEditPage.refreshPanel(). It won't cover the scenario where data is being further processed in the frontend, but probably good enough in most cases.

yesoreyeram · 2024-01-25T11:43:19Z

I'm not sure you would want to assert on content of the data frames in an E2E test. Even though the format is to be considered stable, it's still moving target so you'd need to change your tests whenever a change is introduced. I think we're better of looking at what's being rendered in the table panel

I completely agree. I might not conveyed correctly. I just want to say that I want the table panel content in a format similar to data frames ( doesn't need to be exact data frames ).

yesoreyeram · 2024-01-25T11:46:19Z

If there's a need to assert on the shape of the data frames, they can use the json in the response returned by for example panelEditPage.refreshPanel()

I don't think there is a need to assert the data frame shape. Once we assert the table panel content, that is most sufficient. ( also some datasources does post processing of response in the client side and so in either cases, asserting table panel content is more useful )

yesoreyeram · 2024-01-25T11:47:31Z

How would we be able to get the data frames passed to a panel from within an E2E test? Checking the Query inspector panel data would be slow and brittle.

Definitely I didn't meant to read from query inspector :)

sunker · 2024-01-25T13:06:05Z

Awesome @yesoreyeram - I take it we're on the same page and have a path forward. I'll try to update the PR tomorrow. 👍

sunker · 2024-01-29T15:29:36Z

packages/plugin-e2e/src/models/utils.ts

@@ -0,0 +1,14 @@
+import { Page } from '@playwright/test';
+
+export const radioButtonSetChecked = async (


Planning to expose some kind of UI helpers through the base page, but will do that in a different pr.

sunker · 2024-01-29T15:37:25Z

packages/plugin-e2e/src/models/Panel.ts

+  getData(): Locator {
+    return this.locator().locator('[role="cell"]');
+  }
+


FYI I'm working on adding proper e2e selectors for the table panel in grafana core.

Even though people will probably want to use the Table panel in most cases, it won't cover the scenario when you want to assert on panel (that is not table panel) data in a dashboard, as you can't toggle to Table view from there. Not high priority, but we could easily add parsing functionality to some of the most common panel types. E.g

getData(): Locator { switch (this.visualizationType) { case 'table': return this.locator().locator('[role="cell"]'); case 'logs': return this.locator().locator(selectors.components.Panels.Panel.Logs.logRow); case 'gauge': return this.locator().locator(selectors.components.Panels.Panel.Gauge.value); default: throw new Error(`No data parser implemented for this visualization type: ${this.visualizationType}`); } }

sunker · 2024-01-29T15:48:39Z

Okay @yesoreyeram, code and PR description updated. Would be great if you could take another look.

sunker · 2024-02-08T20:56:03Z

packages/plugin-e2e/src/models/ExplorePage.ts

-    let locator = this.getByTestIdOrAriaLabel(this.ctx.selectors.components.Panels.Panel.title(suffix), {
-      startsWith: true,
-    });
+  private getTablePanelLocator() {


locating table panel and time series panel in explore is a mess. for some reason, selectors seem to change in every version of Grafana. need to look into this at some point and investigate how this can be improved.

yesoreyeram

LGTM. Let's hope we will find more usecases and improve this in later iterations.

grafana-plugins-platform-bot · 2024-02-10T12:35:41Z

🚀 PR was released in @grafana/plugin-e2e@0.13.0 🚀

tolzhabayev assigned sunker Jan 23, 2024

sunker commented Jan 24, 2024

View reviewed changes

sunker requested a review from academo January 24, 2024 07:47

sunker marked this pull request as ready for review January 24, 2024 07:47

sunker requested a review from a team as a code owner January 24, 2024 07:47

sunker requested a review from yesoreyeram January 24, 2024 07:47

sunker mentioned this pull request Jan 24, 2024

Plugin E2E: Tasklist #576

Closed

6 tasks

sunker changed the title ~~[POC] Plugin E2E: Enable assertion on panel data~~ Plugin E2E: Enable assertion on panel data Jan 24, 2024

yesoreyeram reviewed Jan 25, 2024

View reviewed changes

sunker force-pushed the plugin-e2e/table-assertion branch from 95269ad to 61701d4 Compare January 29, 2024 13:09

sunker added minor Increment the minor version when merged release Create a release when this pr is merged labels Jan 29, 2024

sunker commented Jan 29, 2024

View reviewed changes

sunker force-pushed the plugin-e2e/table-assertion branch from f60db74 to 20b0448 Compare February 8, 2024 14:10

enable panel data assertion

596270b

sunker force-pushed the plugin-e2e/table-assertion branch from f99a690 to 596270b Compare February 8, 2024 14:59

sunker added 2 commits February 8, 2024 21:45

revert test

472f006

revert change in workflow file

d5161e3

sunker commented Feb 8, 2024

View reviewed changes

yesoreyeram approved these changes Feb 9, 2024

View reviewed changes

sunker added 2 commits February 10, 2024 06:50

simplity explore

9b4207a

remove redundant func

5df6275

sunker merged commit c81c7a7 into main Feb 10, 2024
20 checks passed

sunker deleted the plugin-e2e/table-assertion branch February 10, 2024 12:30

grafana-plugins-platform-bot bot added the released This issue/pull request has been released. label Feb 10, 2024

sunker mentioned this pull request Mar 20, 2024

Bug: Fix broken selectors in Explore page #728

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin E2E: Enable assertion on panel data #688

Plugin E2E: Enable assertion on panel data #688

sunker commented Jan 23, 2024 •

edited by grafana-plugins-platform-bot bot

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading

sunker Jan 24, 2024

yesoreyeram left a comment

yesoreyeram Jan 25, 2024

sunker Jan 25, 2024

yesoreyeram Jan 25, 2024

yesoreyeram Jan 25, 2024

sunker Jan 25, 2024

yesoreyeram Jan 25, 2024

sunker commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

sunker commented Jan 25, 2024

sunker Jan 29, 2024

sunker Jan 29, 2024

sunker commented Jan 29, 2024

sunker Feb 8, 2024

yesoreyeram left a comment

grafana-plugins-platform-bot bot commented Feb 10, 2024

		@@ -0,0 +1,14 @@
		import { Page } from '@playwright/test';

		export const radioButtonSetChecked = async (

Plugin E2E: Enable assertion on panel data #688

Plugin E2E: Enable assertion on panel data #688

Conversation

sunker commented Jan 23, 2024 • edited by grafana-plugins-platform-bot bot Loading

github-actions bot commented Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

yesoreyeram left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunker commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

yesoreyeram commented Jan 25, 2024

sunker commented Jan 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunker commented Jan 29, 2024

Choose a reason for hiding this comment

yesoreyeram left a comment

Choose a reason for hiding this comment

grafana-plugins-platform-bot bot commented Feb 10, 2024

sunker commented Jan 23, 2024 •

edited by grafana-plugins-platform-bot bot

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading