Feature: Add tags to Cypress tests to enable parallel batch execution in CI without Cy Dashboard

[iamchughmayank]

Description of the Change

Context

This PR proposes adding tags to the Cypress tests to enable selective execution using package @cypress/grep. The aim is to optimise the CI execution time for the test phase by running two groups of tests based on their historic test execution time.

Problem Statement

The job Cypress - Local is taking approximately 20m to execute where the test phase is 17m long. The parallel job of Cypress - EP.io has the similar stats. This increases the wait times for development and reduces dev productivity

Proposal

Cypress offers out-of-the-box solution for parallel execution of tests in CI as well as without. However, both the solutions require subscription of the Dashboard Service. While Dashboard service has a free tier, it does have a limit in that tier (ref)

The solution in this proposal works around it by introducing tags in the tests and using @cypress/grep to selectively run the tests with desired tags.

Using this approach, one can club the tests into multiple batches and run them in parallel using the matrix strategy of Github Actions without using the Dashboard Service.

Implementation

In this PR, I reviewed the tests to find out the tests which were taking longer than the others and I configured them into multiple batches to carefully optimise in such a fashion that parallelisation can be achieved without creating unbalanced test groups.

The tests which were taking longer than others have been tagged with @slow tag and the GitHub workflow file has been updated to run the tests into multiple groups using grep with slow and inverted grep with -slow as per the package documentation

Results

A PoC was done with this approach in the repo Automattic/vip-go-mu-plugins by dividing the tests into three groups which brought down the total execution time from an average of 19m to 8m22s.

I attempted to run similarly in this repo as well, however, I think the tests are broken and showing the results of groups may not be correct representation right now. I can see that several "Scheduled E2E" workflow runs have failed due to erroneous tests.

Feedback

I would love to get feedback on this approach and discuss further on the limitations and benefits of this.

How to test the Change

1. Pull down the PR
2. Follow the steps in tests/cypress/README.md to run tests
3. The tests should run as they are running on develop branch.
4. Next, run the test groups with command npm run cypress:run -- --env grepTags=@slow and npm run cypress:run -- --env grepTags=-@slow
5. The tests should run as they are running on develop branch.

Testing in CI:

It can be tested in multiple ways:

1. Using workflow_dispatch events to run the E2E tests workflow in CI for this branch
2. Use act to run Github workflows locally
3. Create a new branch from this one and add that branch in the push event in cypress-test.yml file. Make a commit in that branch to see the E2E running on push event.

Changelog Entry

Adds tags to the Cypress tests to enable selective execution using package @cypress/grep. The aim is to optimise the CI execution time for the test phase by running two groups of tests based on their historic test execution time.

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added tests to cover my change.
• All new and existing tests pass.