E2E Tests: Add Scheduled job for a Pre-Release Gutenberg block tests

[brbrr]

Changes proposed in this Pull Request:

• Adds a scheduled E2E job to run block tests against pre-release Gutenberg
• Re-organize scripts a bit

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

There is no easy way to test it, unfortunately. To properly test it:

• Checkout this branch and run GUTENBERG=latest ./bin/env.sh start to make sure the plugin setup works as expected
• Create an action in your personal test repo. Something like https://github.com/brbrr/action-test/blob/master/.github/workflows/test-action.yml
• Commit and wait to see some scheduled actions to be triggered
• OR check my test repo, where I did the tests: https://github.com/brbrr/action-test

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ All commits were linted before commit.
  
• 🔴 Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Once your PR is ready for review, check one last time that all required checks (other than "Required review") appearing at the bottom of this PR are passing or skipped.
Then, add the "[Status] Needs Team review" label and ask someone from your team review the code.
Once you’ve done so, switch to the "[Status] Needs Review" label; someone from Jetpack Crew will then review this PR and merge it to be included in the next Jetpack release.

---

Jetpack plugin:

• Next scheduled release: May 4, 2021.
• Scheduled code freeze: April 26, 2021.

[brbrr]

This approach is kinda messy. We would end up with 3 runs for both push and schedule events. All of the "skipped" jobs would build Jetpack and upload artifacts anyway.

[anomiex]

You could do like I did for tests.yml: One lightweight job that produces the matrix, then the matrix job depends on that to use the generated matrix. Too bad GH doesn't have any other way to DRY when you need all the same setup in several similar but not-quite-identical jobs.

Even better would be if the E2E setup and running could be closer to how the rest of the tests do it so we could run them from the same matrix. But that's probably longer term than this PR, as it would probably require dropping the use of wp-env.

[adimoldovan]

I wonder if we really need to run no/latest variation on every PR. With this new pre-release maybe we can only run the GB != bundled in a schedule, and have only a full suite run with the bundled GB in PRs checks. We could then have 2 different workflows.

• Optional we could make the PR workflow depend on the build one, and use the build artefacts as input, so we don't build twice. This will also remove all the PHP setup as we don't need it to run the E2E tests.
• The second workflow will be similar to the one we have now, with a matrix and a Jetpack build, but no event condition.

[ockham]

How about collapsing these two steps into one, and merging their if expressions into one?

    - name: Run blocks tests with ${{ matrix.gutenberg }} Gutenberg
      if: |
        ( github.event_name != 'schedule' && matrix.gutenberg == 'latest' ) ||
        ( github.event_name == 'schedule' && matrix.gutenberg == 'pre-release' )
      working-directory: projects/plugins/jetpack/tests/e2e
      run: yarn test-e2e --testNamePattern=blocks

This approach is kinda messy. We would end up with 3 runs for both push and schedule events. All of the "skipped" jobs would build Jetpack and upload artifacts anyway.

You could build Jetpack in a separate job (without a matrix) and upload the build as an artifact, declare that the test job needs: the build job, and download the artifact there. (We have some precedent for something like that in the Gutenberg repo: See how the create-release job uses the artifact from the build job.)

As for the uploading of the test result artifacts, you could try changing the step's if: from ${{ always() }} to success() || failure() (to maybe skip it if the previous step is skipped?) Not sure if that's going to work, though.

As a further/later optimization, it's probably even possible to also absorb the "run all tests" step into this one (e.g. through the use of an env variable).

[brbrr]

I wonder if we really need to run no/latest variation on every PR. With this new pre-release maybe we can only run the GB != bundled in a schedule, and have only a full suite run with the bundled GB in PRs checks.

Yeah, I don't think we really need to run latest GB tests on every PR. We probably can get rid of the matrix in the job altogether.

Moving the build step into a separate job is a good idea. I'll look into it in a follow up PR

[brbrr]

with the latest changes from @adimoldovan around E2E workflow, I made the matrix defined in runtime. During the scheduled run, an additional job would be triggered that would run block tests with the latest GB plugin.

[anomiex]

You could do like I did for tests.yml: One lightweight job that produces the matrix, then the matrix job depends on that to use the generated matrix. Too bad GH doesn't have any other way to DRY when you need all the same setup in several similar but not-quite-identical jobs.

Even better would be if the E2E setup and running could be closer to how the rest of the tests do it so we could run them from the same matrix. But that's probably longer term than this PR, as it would probably require dropping the use of wp-env.

[fullofcaffeine]

Here's another idea, let me know if it makes sense to you: What if we automatically trigger the E2E tests as soon as a new version of GB is released (in addition to triggering them every 12hr)?

When it comes to GB Calypso E2E tests, we can't yet do this, because we still need to manually update the plugin in the SVN for edge sites. But here, the whole process of installing and activating the plugin is automatic.

This would be useful to have the results as soon as possible after a given release, and would allow JP teams to act on them more readily. We (@Automattic/team-calypso) would also be following the test results on Slack, and try to coordinate the fixes as soon as humanly possible.

Unfortunately, there doesn't seem to be a way to "listen" to releases from other repos using GH actions, but maybe we could use the cron strategy described here.

cc @ockham

[brbrr]

@fullofcaffeine this is an interesting idea. Although, I don't think it is possible to implement solely within Jetpack's CI configuration. The problem here is that to identify a new release we need to compare it against something. For that, we would need to store the current release somewhere. With GH Actions we can't do that. I have a few possible solutions though:

1. Set up a web server that would monitor Gutenberg releases, store the current release, and trigger a GH Action for Jetpack when a new one is out.
2. Add a webhook into a Gutenberg release, which would trigger a GH Action in a Jetpack repo. This one would probably need an "actor" of some sort to actually trigger the run in Jetpack.
3. As suggested in SO thread, but instead of relying on a file with the "current" version, we can look for a release date, and see if it was released after the previously scheduled run. This one would be tricky as well since GH does not guarantee that scheduled actions would be triggered precisely "on time"

[anomiex]

In case you didn't see it yet, either of the "trigger a GH Action" methods would make use of https://docs.github.com/en/rest/reference/repos#create-a-repository-dispatch-event, which needs to be invoked using a token with the "repo" scope.

As for storing data, you could store it in a gist. I see marketplace actions exist for reading and writing them. Or you could create a special branch that contains a metadata file instead of code.

[fullofcaffeine]

mysqlcheck: Got error: 1045: Access denied for user 'root'@'172.31.0.6' (using password: YES) when trying to connect

@fullofcaffeine is probably related to outdated cached Docker images. try dropping/updating your wordpress image (not the container)

This was the culprit: WordPress/gutenberg#29847. This branch has @wordpress/env @ 4.0.0, which doesn't have the fix, master already has 4.0.2 setup. I'll rebase.

[fullofcaffeine]

This was the culprit: WordPress/gutenberg#29847. This branch has @wordpress/env @ 4.0.0, which doesn't have the fix, master already has 4.0.2 setup. I'll rebase.

Wrote too fast. Still can't get it to work, even after removing all images, updating from master, re-installing all modules. No idea what's wrong. Does it work for you @brbrr?

That being said, that seems to be wp-env-related. Assuming this works well in CI, then I'd say let's get this shipped and I'll help testing by watching the slack channel as new GB versions are shipped.

[adimoldovan]

Just to make sure the behaviour I understand from this code is the intended one:

• On schedule all tests will run
• On pull request the gutenberg tests (with plugin set up) will not run

[brbrr]

Yes, but I'd frame it like this:

• For pull requests, we run all the tests with bundled Gutenberg.
• For the scheduled job, we run all the tests with bundled Gutenberg PLUS blocks tests with the latest dev Gutenberg

[adimoldovan]

This looks good to me and I say let's ship it.
It looks like it needs a master merge or rebase first.

[fullofcaffeine]

Thanks for working on this, @brbrr! ❤️