Plugin E2E: Change FE feature toggles during test session

[sunker]

What this PR does / why we need it:

This is a bit experimental, and I'm not sure we want to go down this road so feedback is much appreciated.

I foresee that many plugin authors are going to want to test their plugins with certain feature toggles both enabled and disabled. Currently, the only way to change the value of a feature toggle in CI is by setting the GF_FEATURE_TOGGLES_ENABLE environment variable when starting the grafana container in the github workflow. Then they can use the isFeatureToggleEnabled fixture inside a test to check whether a certain feature is enabled. This works, but in order to try different values for a certain feature, they'll have to start multiple versions of Grafana in the workflow. This is complicated and it will also demand more resources.

As you know, Grafana feature toggles are being passed from Grafana backend to the frontend by setting window.grafanaBootData.settings.featureToggles. Playwright provide some guidelines for how to mock browser apis, such as the window object. In this PR, I'm adding a addInitScript function that is executed whenever the page is navigated. The function will poll for the existence of window.grafanaBootData, and when it exists it will override and/or add features that are defined in the playwright test session. The plugin-e2e consumer can then define their feature toggles in the playwright.config.ts like this:

  use: {
    baseURL: 'http://localhost:3000',
    featureToggles: {
      redshiftAsyncQueryDataSupport: false,
    },
    ...
  },

Of override the feature toggle for a certain file:

#queryEditor.spec.ts
test.use({
  featureToggles: {
    redshiftAsyncQueryDataSupport: true,
  },
});

I know this solution is kind of brittle and I wouldn't raise the PR if it wasn't for the fact the problem feels important to solve.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

📦 Published PR as canary version: Canary Versions

✨ Test out this PR locally via:

npm install @grafana/create-plugin@3.2.0-canary.651.469ce0d.0
npm install @grafana/plugin-e2e@0.10.0-canary.651.469ce0d.0
# or 
yarn add @grafana/create-plugin@3.2.0-canary.651.469ce0d.0
yarn add @grafana/plugin-e2e@0.10.0-canary.651.469ce0d.0

[github-actions]

Hello! 👋 This repository uses Auto for releasing packages using PR labels.

✨ This PR can be merged and will trigger a new minor release.
NOTE: When merging a PR with the release label please avoid merging another PR. For further information see here.

[academo]

@sunker We have a mechanism to enable and disable feature toggles in frontend already that uses local storage. you can see how it works here so there's no need to write code that modifies the grafanaBootData https://github.com/grafana/grafana/blob/4fa6bad7c03948cd648f784283bc60d7b2199ad3/packages/grafana-runtime/src/config.ts#L217

[sunker]

@sunker We have a mechanism to enable and disable feature toggles in frontend already that uses local storage. you can see how it works here so there's no need to write code that modifies the grafanaBootData https://github.com/grafana/grafana/blob/4fa6bad7c03948cd648f784283bc60d7b2199ad3/packages/grafana-runtime/src/config.ts#L217

Oh I did not know that. Thanks @academo, I'll try that instead.

[mckn]

I think the need for something like this is valid and something that we should provide! The question is how we should tailor this. If we don't change the value across the entire stack (frontend + backend) there is a risk of having functionality that doesn't function properly.

On the other hand, if we update the value across the entire stack we might have problems running tests affected by the same toggle in parallel.

I guess there are pros/cons with every approach. Can we do something like the following?

• Setup the Grafana instance being used during E2E tests to allow feature toggles to be changed during the runtime.
• Use the following API to update the feature toggle prior to navigating to the page you want to test (https://github.com/grafana/grafana/blob/main/pkg/api/featuremgmt.go#L48)
• Group tests that affects a feature flag and recommend people not to run them in parallel. I think this can be configured like this. https://playwright.dev/docs/test-parallel#parallelize-tests-in-a-single-file
• Reset the feature flag after the test has been run to the original value.

Thoughts on that? This approach will update the feature flag

[sunker]

Thanks for great input @mckn!

• Use the following API to update the feature toggle prior to navigating to the page you want to test (https://github.com/grafana/grafana/blob/main/pkg/api/featuremgmt.go#L48)

Great, I did not know we had this API either.

One concern that I have with using this API is that it was introduced in Grafana 10.2.0. So far, all features in plugin-e2e are backwards compatible to Grafana 8.3.0. We could probably limit this feature in plugin-e2e to Grafana versions >=10.2.0, but it would not be ideal. Actually the same issue applies to using the overrideFeatureTogglesFromLocalStorage method @academo as it was introduced in Grafana 10.1.0.

On the other hand, if we update the value across the entire stack we might have problems running tests affected by the same toggle in parallel.

I think this is a very valid concern. Just to circumvent this problem, I wonder if it's best to limit this capability in plugin-e2e to only change the frontend part of feature toggle. Maybe it's fine as long as we are very clear about that in the docs? If a plugin authors needs to tests a feature toggle across the entire stack, it needs to be defined when configuring the Grafana instance.

[mckn]

Thanks for great input @mckn!

• Use the following API to update the feature toggle prior to navigating to the page you want to test (https://github.com/grafana/grafana/blob/main/pkg/api/featuremgmt.go#L48)

Great, I did not know we had this API either.

One concern that I have with using this API is that it was introduced in Grafana 10.2.0. So far, all features in plugin-e2e are backwards compatible to Grafana 8.3.0. We could probably limit this feature in plugin-e2e to Grafana versions >=10.2.0, but it would not be ideal. Actually the same issue applies to using the overrideFeatureTogglesFromLocalStorage method @academo as it was introduced in Grafana 10.1.0.

On the other hand, if we update the value across the entire stack we might have problems running tests affected by the same toggle in parallel.

I think this is a very valid concern. Just to circumvent this problem, I wonder if it's best to limit this capability in plugin-e2e to only change the frontend part of feature toggle. Maybe it's fine as long as we are very clear about that in the docs? If a plugin authors needs to tests a feature toggle across the entire stack, it needs to be defined when configuring the Grafana instance.

That would work! An alternative would be to only do the frontend part of the toggle in versions earlier than 10.2.0? If that would be a way forward.

[sunker]

An alternative would be to only do the frontend part of the toggle in versions earlier than 10.2.0? If that would be a way forward.

Think it could become a bit confusing. Also, then they may have to write different test code for different versions of Grafana, which is not ideal.

[sunker]

@academo this PR should be ready for review now. Now I'm updating local storage in case Grafana version => 10.1.0. If not, I'm updating the bootData directly.

[sunker]

I think the js in this file should be 100% compatible with recent versions of all browsers.