Bohandley/integration tests #27

bohandley · 2024-02-26T17:29:38Z

Use playwright and @grafana/plugin-e2e to write tests in parity with core Grafana prometheus.
Create github workflow with docker-compose-debug.yaml to test integration with Prometheus

How to test:

Build the backend mage -v
Build the frontend yarn install yarn build
run docker compose -f docker-compose-debug.yaml up for Grafana at locahost:3099 and Prometheus at localhost:9090
run yarn test:e2e to run plugin-e2e tests against the docker Grafana instance and Prometheus.

Next steps

Write tests for auth to test middleware which needs to be added in plugin-sdk

…other running grafana instances

…g off basic auth in grafana docker

NWRichmond · 2024-03-11T19:31:34Z

.github/workflows/e2e.yml

It might be worth updating to actions/checkout@v4, actions/setup-go@v5, and actions/setup-node@v4. The latest versions of the Go and Node actions have the most capabilities with respect to dependency caching, which can significantly speed up CI/CD.

For more info:

Go: https://github.com/actions/setup-go#caching-dependency-files-and-build-outputs

Node: https://github.com/actions/setup-node?tab=readme-ov-file#caching-global-packages-data

An important difference between the latest Go & Node actions is that the Go one enables caching by default, and the Node one does not - it requires a few lines of config.

Awesome, I'm going to add this as an issue to improve CI/CD for this repo!

Go caching is working! Saved 6 minutes of workflow time. I'm a little stuck on the node caching currently but it is captured in the issue.

sunker

I think this looks great @bohandley! Added a few comments with some tips and tricks related to playwright and plugin-e2e. Let me know if you have any questions. 👍

sunker · 2024-03-12T13:56:28Z

.github/workflows/e2e.yml

+      - name: Wait for prometheus to be configured in grafana
+        uses: nev7n/wait_for_response@v1
+        with:
+          url: 'http://localhost:3099/api/datasources/uid/prometheus-amazon/health'


Wouldn't just waiting for this url be enough?

I'm not sure what you mean? Waiting for this response ensures that the Grafana can make a request to Prometheus and the response is a 200. This might not be necessary though, waiting for Prometheus and Grafana might be enough.

Yeah I mean if this url returns 200, it means Grafana and Prometheus are running so we can remove those steps in the workflow?

sunker · 2024-03-12T14:02:10Z

.github/workflows/e2e.yml

+      - name: Start the docker container for E2E
+        run: |
+          docker compose -f docker-compose-debug.yaml pull
+          docker compose -f docker-compose-debug.yaml up -d


If I understand correctly, this PR only tests your plugin against the latest enterprise main build. I see you have specified grafana 10.4.0 as min dep in the plugin.json. If the plugin is supposed to be compatible with all grafana versions beyond 10.4.0, I suggest you start using the e2e-versions Action. This Action will resolve a range of Grafana versions to test against so you can be sure you plugin is compatible with that entire range. It's documented here.

This is awesome! We still need to decide what versions we will support so I will add this to an issue to update the workflow.

sunker · 2024-03-12T14:05:42Z

.github/workflows/e2e.yml

+        with:
+          name: test-results
+          path: test-results/
+          retention-days: 30


Downloading the report manually and firing it up locally is pretty time consuming. The example workflow for a backend ds plugin in this guide has a step that pushes the playwright report to a GCS bucket. It also drops a link to the report in the workflow summary, so you can browse the report directly from the PR without having to download it manually.

This is super helpful. We will have to choose a storage location appropriate for the team that will take this repository on, so I will add this as an issue to improve the debugging experience.

This is super helpful. We will have to choose a storage location appropriate for the team that will take this repository on, so I will add this as an issue to improve the debugging experience.

If you use the publish-report Action, you won't be able to choose storage location. It uses Grafana's dev artifact storage bucket, and the url to the report will be:
https://storage.googleapis.com/releng-pipeline-artifacts-dev/<repo-name>/<pr-number>/<the-version-of-grafana-you're-testing-against>/

sunker · 2024-03-12T14:07:00Z

e2e/annotation-editor.spec.ts

+
+    await annotationEditPage.datasource.set(ds.name);
+    await annotationEditPage
+      .getByTestIdOrAriaLabel(selectors.components.DataSource.Prometheus.queryEditor.code.queryField).focus();


Don't think you need this

You are right, I don't think I need the focus or setting the DS, because the default DS is Prometheus.

sunker · 2024-03-12T14:07:43Z

e2e/configuration.spec.ts

+    page,
+  }) => {
+    const ds = await readProvisionedDataSource<DataSourcePluginOptionsEditorProps<PromOptions>>({ fileName: 'datasources.yml' });
+    const configPage = await createDataSourceConfigPage({type: ds.type, deleteDataSourceAfterTest: true});


deleteDataSourceAfterTest defaults to true, so you can remove this. 👍

sunker · 2024-03-12T14:23:17Z

e2e/query-editor.spec.ts

+    await explorePage.datasource.set(ds.name);
+    // query patterns
+    await expect(explorePage
+      .getByTestIdOrAriaLabel(selectors.components.QueryBuilder.queryPatterns)).toBeVisible();


Using the selectors from grafana/e2e-selectors is probably not a good idea here. If the value of this selector changes in for example 11.0.0, your test will no longer work. I suggest one of the following:

define the selector locally in this plugin repo

if the same selector should be used by multiple plugins, define a versioned selector in plugin-e2e instead and use the selector fixture in your test.

This will require an issue created for core Grafana. We are currently using the same selectors for core Prometheus DS which still relies on cypress. Additional question: can we use plugin-e2e in core Grafana for core data sources?

Future work will hopefully only use one set of selectors and one framework for e2e tests, and as we are keeping all of these Prometheus data sources as similar as possible, I will create an issue for this.

Additional question: can we use plugin-e2e in core Grafana for core data sources?

Yes! Just add a new folder with your plugin name in this directory, and then setup a new project that runs your tests here. I'd be happy to help out!

Future work will hopefully only use one set of selectors and one framework for e2e tests, and as we are keeping all of these Prometheus data sources as similar as possible, I will create an issue for this.

Our long term goal is to detach @grafana/e2e-selectors from grafana packages so that it's no longer tied to a certain Grafana version. Instead, each selector will be a valid from a min Grafana version (that's how it works in plugin-e2e today). e.g

Components: { CodeEditor: { container: { '10.2.3': 'data-testid Code editor container', '8.3.0': 'Code editor container', }, }, ... }

I began an epic describing the changes in Grafana that would need to happen to use selectors across all the Prometheus data sources, core and external. If you have any suggestions or improvements to the epic please feel free to edit the description!

grafana/grafana#84394

sunker · 2024-03-12T14:25:45Z

e2e/query-editor.spec.ts

+
+      // await explorePage.datasource.set(dsDefaultEditorBuilder.name);
+
+      await page.getByTestId('data-testid Select a data source').click();


I will troubleshoot this. If you see this error in ci again, please share playwright report.

sunker · 2024-03-12T14:29:07Z

package.json

    "@grafana/eslint-config": "6.0.1",
+    "@grafana/plugin-e2e": "0.17.1",


nit: the most recent version is 0.19.0

sunker · 2024-03-12T14:32:09Z

playwright.config.ts

+  projects: [
+    {
+      name: 'data source',
+      use: { ...devices['Desktop Chrome']},


Up to you, but I don't recommend testing with anonymous user. See this guide on how to add a setup project that ensures all tests will start as logged in admin user. It won't hurt performance in your tests.

sunker · 2024-03-12T14:33:29Z

.gitignore

+.eslintcache
+/test-results/
+# /playwright-report/


I suggest you uncomment this - there's not reason to version control you local generated reports.
Also add /playwright/.auth/

bohandley · 2024-03-13T19:11:49Z

.github/workflows/e2e.yml

+      #   with:
+      #     version: latest
+      #     args: coverage
+      - name: Build backend


Building the backend here @itsmylife

bohandley · 2024-03-13T19:12:07Z

.github/workflows/e2e.yml

+          cache-dependency-path: '**/package-lock.json'
+      - name: Install dependencies
+        run: npm install -g yarn && yarn install --frozen-lockfile;
+      - name: Build Frontend


Building the frontend here @itsmylife

itsmylife · 2024-03-13T19:28:40Z

docker-compose.yaml

@@ -12,4 +12,4 @@ services:
      - 3000:3000/tcp


Shouldn't this to be "3000:3000"
with quotes around and without tcp?

Oh! That docker file is not what we use to test things currently, but it comes "out of the box" with a backend plugin.

The docker compose file with all the nice things is docker-compose-debug.yaml.

NWRichmond

The expected tests are passing when I run the pull_request action with act.

Nicely done, @bohandley!

bohandley added 30 commits February 21, 2024 14:33

update dependencies and add plugin-e2e and playwright

ca63d5d

fix plugin.json for missing property

7bb3bb9

remove unnecessary @ts-ignore

7fa95b1

add plugin-e2e and playwright

6eb91cb

add docker debug file with provisioning

8c87d93

update provisioning for docker

748f079

update grafana deps with actual versions

2a40343

update provisioning for plugin-e2e tests

8c6c624

begin setting up integration tests

ba04e71

update README with instructions for e2e tests

4cb61f0

update docker debug to use grafana port that does not interfere with …

b7d7691

…other running grafana instances

remove playwright auth dependency as not needed because we are turnin…

987f7c9

…g off basic auth in grafana docker

add script for playwright e2e tests

dd9b24b

get playwright working and update readme

93255e4

add first configuration test

7f3c6d7

update config spec e2e tests

d9f56b0

make tests work correctly and test correctly

0141329

remove unused imports

55ff0cb

clean comments

514ecf4

update to latest version of prom

b7865c3

add special data source provisioning file for testing config

e0e3cbe

rewrite cypress tests in playwright for query editor

6506c6f

remove unused data import

fb73140

add tests for variable query editor

3a80071

add e2e test for annotation editor

2c6f5fa

removed unnecessary test

34b0c1d

add dashboard json for future tests

78d3108

begin testing e2e workflow

55264de

fix workflow for broken tests

4a43717

simplify workflow for only testing e2e tests

5109b31

bohandley added 5 commits March 11, 2024 12:06

add e2e video storage

53cb73e

use async await for setting data source

79b889a

retain videos on failure

64e2497

test with explicit navigation to explore page

2acd402

explicitly change the data source picker

356176b

NWRichmond reviewed Mar 11, 2024

View reviewed changes

Base automatically changed from bohandley/use-prom-library-with-sigv4-config to main March 12, 2024 13:34

update checkout and setup go to use caching

297860e

sunker reviewed Mar 12, 2024

View reviewed changes

bohandley added 11 commits March 12, 2024 09:43

remove reports from git

0576175

remove unnecessary code

9412e6a

remove deleteDataSourceAfterTest and use toHaveAlert in config tests

b68ceb6

update plugin-e2e version

8e3b0cd

test with plugin-e2e auth instead of anonymous admin

62955e1

remove await from autowaiting actions

e6cb8bf

set default timeout to 10s for tests

6af3010

remove unused code

673e9b8

try fixing caching

b86b09a

fix caching

9b7df1a

remove healthcheck because anonymous auth was removed

ef48b54

bohandley mentioned this pull request Mar 13, 2024

Prometheus: Improve integration tests with plugin-e2e grafana/grafana#84394

Open

4 tasks

yarn cache workflow

f53f785

bohandley commented Mar 13, 2024

View reviewed changes

yarn cache workflow fix

6440d59

itsmylife reviewed Mar 13, 2024

View reviewed changes

itsmylife approved these changes Mar 13, 2024

View reviewed changes

NWRichmond approved these changes Mar 13, 2024

View reviewed changes

bohandley merged commit b344d4e into main Mar 13, 2024
2 checks passed

bohandley deleted the bohandley/integration-tests branch March 13, 2024 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bohandley/integration tests #27

Bohandley/integration tests #27

bohandley commented Feb 26, 2024 •

edited

Loading

NWRichmond Mar 11, 2024

bohandley Mar 12, 2024

bohandley Mar 12, 2024

bohandley Mar 12, 2024

sunker left a comment

sunker Mar 12, 2024

bohandley Mar 12, 2024

sunker Mar 13, 2024

sunker Mar 12, 2024

bohandley Mar 12, 2024

bohandley Mar 12, 2024

sunker Mar 12, 2024

bohandley Mar 12, 2024

bohandley Mar 12, 2024

sunker Mar 13, 2024

sunker Mar 12, 2024

bohandley Mar 12, 2024

sunker Mar 12, 2024

sunker Mar 12, 2024 •

edited

Loading

bohandley Mar 12, 2024

bohandley Mar 12, 2024

sunker Mar 13, 2024

bohandley Mar 13, 2024

sunker Mar 12, 2024

sunker Mar 12, 2024

sunker Mar 12, 2024

sunker Mar 12, 2024

bohandley Mar 13, 2024

bohandley Mar 13, 2024

itsmylife Mar 13, 2024

bohandley Mar 13, 2024

NWRichmond left a comment


		// await explorePage.datasource.set(dsDefaultEditorBuilder.name);

		await page.getByTestId('data-testid Select a data source').click();

		"@grafana/eslint-config": "6.0.1",
		"@grafana/plugin-e2e": "0.17.1",

Bohandley/integration tests #27

Bohandley/integration tests #27

Conversation

bohandley commented Feb 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunker Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NWRichmond left a comment

Choose a reason for hiding this comment

bohandley commented Feb 26, 2024 •

edited

Loading

sunker Mar 12, 2024 •

edited

Loading