Execute E2E tests more frequently #3499
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
So far, we've been running E2E tests only when a PR was changing feature flag, contained `e2e` in the name of the feature branch or got merged to `main`. This prevented us from noticing changes braking E2E tests fast enough - we sometimes were noticing such changes only after they got already merged to `main`. In this commit we modify CI config in a way that most of the E2E tests will be run on every PR update. We also allow for specifying tests which shouldn't be run this often - such tests should be marked with `@slow` tag in the test name (read more here: https://playwright.dev/docs/test-annotations#tag-tests). We may want to use this feature when we'll have tests that require spending some funds.
michalinacienciala
force-pushed
the
run-e2e-tests-more-frequently
branch
from
3b9fa3e
to
168d945
Compare
When no `@slow`-tagged tests are found, the step is failing.
Shadowfiend
approved these changes
Jun 26, 2023
Shadowfiend
approved these changes
Jun 27, 2023
| - name: Run Playwright tests | ||
| run: xvfb-run npx playwright test | ||
| # Some tests that we have configured in the `e2e-tests` folder may require | ||
| # spending funds. Although they're desined to not spend much, with |
There was a problem hiding this comment.
Suggested change
| # spending funds. Although they're desined to not spend much, with | |
| # spending funds. Although they're designed to not spend much, with |
| # frequent execution that can accumulate. We don't want to execute such | ||
| # tests on every PR update. We'll tag those tests with the `@slow` tag. | ||
| - name: Run free Playwright tests | ||
| run: xvfb-run npx playwright test --grep-invert @slow |
There was a problem hiding this comment.
Let's use yarn instead of npx for consistency/simplicity across the repo.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So far, we've been running E2E tests only when a PR was changing feature flag, contained
e2ein the name of the feature branch or got merged tomain. This prevented us from noticing changes braking E2E tests fast enough - we sometimes were noticing such changes only after they got already merged tomain. In this commit we modify CI config in a way that most of the E2E tests will be run on every PR update. We also allow for specifying tests which shouldn't be run this often - such tests should be marked with@slowtag in the test name (read more here: https://playwright.dev/docs/test-annotations#tag-tests). We may want to use this feature when we'll have tests that require spending some funds.Note: depending on which PR gets merged first, we may need to update either this PR or #3418.
Latest build: extension-builds-3499 (as of Tue, 27 Jun 2023 01:23:53 GMT).