Conditionally run E2E tests

[pierlon]

Summary

Fixes currently failing E2E tests.

The test AMP Settings Screen › should not allow AMP Stories to be enabled when Gutenberg is not active failed because the condition that determines the status of the Stories experience checkbox now returns true on WP >= 5.3. Specifically, the test fails due to the checkbox no longer being disabled.

My proposed solution is to conditionally run tests based on the given environment. In this PR, I've made a wrapper for the test-e2e script (provided through wp-scripts) that checks if the current WordPress version is greater than or equal to 5.3, and if so, ignores the failing test. The wrapper is written with extensibility in mind for future cases where ignoring a test within a given constraint may be necessary.

Checklist

• My pull request is addressing an open issue (please create one otherwise).
• My code is tested and passes existing tests.
• My code follows the Engineering Guidelines (updates are often made to the guidelines, check it out periodically).

[westonruter]

I'm very inexperienced with E2E testing, but instead of creating this wrapper is not something like this possible:

diff --git a/tests/e2e/specs/amp-options.js b/tests/e2e/specs/amp-options.js
index 237fea8f..b874a08a 100644
--- a/tests/e2e/specs/amp-options.js
+++ b/tests/e2e/specs/amp-options.js
@@ -1,3 +1,8 @@
+/**
+ * External dependencies
+ */
+const semver = require( 'semver' );
+
 /**
  * WordPress dependencies
  */
@@ -54,6 +59,10 @@ describe( 'AMP Settings Screen', () => {
 	} );
 
 	it( 'should not allow AMP Stories to be enabled when Gutenberg is not active', async () => {
+		if ( semver.gte( '5.3.0', semver.coerce( process.env.WP_VERSION ) ) ) {
+			return;
+		}
+
 		await deactivatePlugin( 'gutenberg' );
 
 		await visitAdminPage( 'admin.php', 'page=amp-options' );

[westonruter]

Or rather:

diff --git a/tests/e2e/specs/amp-options.js b/tests/e2e/specs/amp-options.js
index 237fea8f..eeb1ad2a 100644
--- a/tests/e2e/specs/amp-options.js
+++ b/tests/e2e/specs/amp-options.js
@@ -1,3 +1,8 @@
+/**
+ * External dependencies
+ */
+const semver = require( 'semver' );
+
 /**
  * WordPress dependencies
  */
@@ -53,7 +58,9 @@ describe( 'AMP Settings Screen', () => {
 		expect( await page.$eval( '#amp-settings', ( el ) => el.matches( ':invalid' ) ) ).toBe( true );
 	} );
 
+	if ( semver.lt( '5.3.0', semver.coerce( process.env.WP_VERSION ) ) ) {
 		it( 'should not allow AMP Stories to be enabled when Gutenberg is not active', async () => {
+
 			await deactivatePlugin( 'gutenberg' );
 
 			await visitAdminPage( 'admin.php', 'page=amp-options' );
@@ -64,6 +71,7 @@ describe( 'AMP Settings Screen', () => {
 
 			await activatePlugin( 'gutenberg' );
 		} );
+	}
 
 	it( 'should allow AMP Stories to be enabled when Gutenberg is active', async () => {
 		await visitAdminPage( 'admin.php', 'page=amp-options' );

[pierlon]

As far as I'm concerned the WP version isn't provided in the environment.

[westonruter]

This works. I'd like to get Pascal's thoughts when he's available.