Add E2E testing using Cypress

[dinhtungdu]

Description of the Change

This PR:

• Sets up E2E testing using Cypress.
• Adds test cases for Podcasts taxonomy and Block editor.
• Uses GitHub Actions to:
  • Run the test on PRs, develop, and trunk.
  • Test against WordPress's latest development, production, and minimum version.
  • For PRs, the test runs only on the latest stable version for faster execution.
  • When the PR is merged to production, additional core versions are tested.

Alternate Designs

See #116. The reason to choose Cypress over WP E2E testing is explained in #118 (comment)

Applicable Issues

#118

Changelog Entry

Added: E2E testing using Cypress and WordPress Evn.

[markjaquith]

Your wp-env GitHub Actions setup here saved me some Googling, so here is some Cypress advice in thanks!

[markjaquith]

You can put http://localhost:8889 into cypress.json under baseUrl and then you can just supply a URL starting with the path (or just use your visitAdminPage() Cypress command).

[dinhtungdu]

@markjaquith Thank you! I refactored this to use visitAdminPage. I put the test domain to cypress.json file before but I removed it in my latest commit. Instead of using hard-coded value, I get it dynamically from wp-env. I do it that way so I can run the test against multiple WP versions. Because you're using wp-env too, it may be useful for you.

Ps: I actually followed your snippet on cypress-io/cypress#909 (comment).

[markjaquith]

@dinhtungdu Nice!

I had used a plugin to set the WP username and password if not overridden in cypress.env.json, but I hadn't thought to pull the baseUrl from there.

	if (!config.env.wp_username) {
		config.env.wp_username = 'admin';
		config.env.wp_password = 'password';
	}

[markjaquith]

Doing conditional testing is not recommended in Cypress. It can lead to really brittle tests. For instance, if WordPress started rendering the plugin deactivation buttons with JavaScript, you might check for its presence before it's there. You can end up with weird race conditions where the tests fail on a slow machine, or fail on a fast machine.

Instead of doing conditional testing when you're unsure of the database's state before the test is run, make it so beforehand.

So, before you cy.visit() the plugins page, you could run this:

cy.exec('npm run env run tests-cli wp plugin deactivate simple-podcasting')

And then you you can just do cy.get('#deactivate-simple-podcasting').click();

[youknowriad]

cy.exec('npm run env run tests-cli wp plugin deactivate simple-podcasting')

This adds a dependency to the environment (access to the environment). Often it's better to be able to run e2e tests against a given URL without knowing how the environment is actually prepared (docker, local, VM...). This allows running the same suite of tests against multiple environments, comparison... It might not be a requirement here but it's one that can be important sometimes so I just wanted to raise it.

[markjaquith]

Yep, that's a good thing to keep in mind. You could also abstract it into cy.deactivatePlugin('simple-podcasting') and handle environment differences there. I'm just wary of conditional code in UI tests, because it means you're going into the test with uncertainty about what you should be encountering, and repeat runs of the test could take different paths and make different assertions.

[dinhtungdu]

Thanks for the head-up! This test doesn't need conditional testing actually. The plugin will be activated if the environment is set up correctly. For this case, it's safe to assume that the plugin is activated because the test relies on wp-env.

[helen]

Made an issue to track various efforts and the overall conversation in #118

[dinhtungdu]

@helen @jeffpaul This PR is ready to review/merge.