Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update/e2e tests options #14129

Merged
merged 9 commits into from
Mar 15, 2019
Merged

Update/e2e tests options #14129

merged 9 commits into from
Mar 15, 2019

Conversation

draganescu
Copy link
Contributor

Description

Following #13993 and part of #8319 this adds the option to run all e2e test commands with an extra interactive commands, which shows Chrome at a human speed, so you can visually inspect your test.

So now we can run:

npm run test-e2e interactive // all tests interactively
npm run test-e2e interactive FILE_NAME // one test file interactively
npm run test-e2e:watch interactive // all tests interactively and watch for changes
npm run test-e2e:watch interactive FILE_NAME // one test file interactively and watch for changes

How has this been tested?

Ran locally with the interactive command and appears to work as expected.

Note

@talldan suggested we'd have

npm run test-e2e:watch --interactive

but the only way to have that is if we'd also append more '--' as in

npm run test-e2e:watch -- --interactive

and I figured it'd be too many dashes, plus when passed to jest it would throw an unrecognised arg error so we'd have clean it before.

Just using the word seemed to work out of the box, maybe ;)

@draganescu draganescu added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Feb 26, 2019
@draganescu draganescu self-assigned this Feb 26, 2019
@draganescu draganescu added the [Type] Enhancement A suggestion for improvement. label Feb 26, 2019
@draganescu draganescu added this to the Documentation & Handbook milestone Feb 26, 2019
@draganescu
Copy link
Contributor Author

@gziolo travis is failing again, would my change break anything other than this simple command?

@ntwb
Copy link
Member

ntwb commented Feb 27, 2019

I've restarted the tests, failures appear unrelated to this PR

@talldan
Copy link
Contributor

talldan commented Feb 28, 2019

I had to restart the one test again—passing now!

@aduth
Copy link
Member

aduth commented Feb 28, 2019

but the only way to have that is if we'd also append more '--' as in

This is consistent though with other usage / recommendations though:

https://github.com/WordPress/gutenberg/blob/master/docs/contributors/testing-overview.md#tldr-broken-snapshots

gutenberg/package.json

Lines 188 to 189 in fe2136d

"test-unit:update": "npm run test-unit -- --updateSnapshot",
"test-unit:watch": "npm run test-unit -- --watch",

Personally, if it's an argument, it feels more appropriate to use a standard arguments syntax. It feels a bit awkward initially, but this is the standard way of passing arguments to the underlying command of any npm script.

Related: npm/npm#5518

plus when passed to jest it would throw an unrecognised arg error so we'd have clean it before

It's an interesting point, since we have options supported by our abstraction, and options to pass through to Jest. Arguably, we should be explicit with our interface in such a way that we only pass through arguments either formed in our script handler, or whitelisted to be handled by Jest. But this is already at odds with how we use the test scripts, where as seen above we pass through Jest's --updateSnapshot and --watch arguments.

I don't really have a good solution on mind at the moment.

@gziolo
Copy link
Member

gziolo commented Mar 4, 2019

npm run test-e2e:watch -- --interactive

Yes, this is the way how it is handled with npm. At the same time you can always use:

./node_modules/.bin/wp-scripts test-e2e --interactive

and everything is going to work as expected.

Copy link
Member

@gziolo gziolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't work the way it was proposed:

> gutenberg@5.1.1 test-e2e /Users/gziolo/PhpstormProjects/gutenberg
> wp-scripts test-e2e --config packages/e2e-tests/jest.config.js "interactive"

No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0
In /Users/gziolo/PhpstormProjects/gutenberg/packages/e2e-tests
  250 files checked.
  testMatch: **/__tests__/**/*.js,**/?(*.)(spec|test).js,**/test/*.js - 62 matches
  testPathIgnorePatterns: /node_modules/ - 250 matches
  testRegex:  - 250 matches
Pattern: interactive - 0 matches

The issue is that interactive is passed as a regular CLI param and Jest processes it as it was file pattern.

The only way to make it work is what @aduth shared:

Arguably, we should be explicit with our interface in such a way that we only pass through arguments either formed in our script handler, or whitelisted to be handled by Jest. But this is already at odds with how we use the test scripts, where as seen above we pass through Jest's --updateSnapshot and --watch arguments.

This command integrates both Jest and Puppeteer so it becomes a bit complex in comparison to all other commands. One way of making it easier to whitelist would be to prefix all custom arguments with --puppeteer-.

@draganescu
Copy link
Contributor Author

draganescu commented Mar 7, 2019

@gziolo I have implemented the argument as @aduth suggested with the extra dashes. Now the command can optionally be run as:

npm run test-e2e:watch -- --puppeteer-interactive --puppeteer-slowmo=10

Also, I left the speed at 80 as a default, and also, see above, added a new argument for changing the speed. To solve the problems of passing all new arguments to Jest I added a small new function that removes arguments with certain prefixes from the list of arguments that is passed to Jest.

@draganescu
Copy link
Contributor Author

draganescu commented Mar 7, 2019

Another option would to not filter by prefix, but instead by command name. This way we would have shorter arguments for example:

npm run test-e2e:watch -- --interactive --slowmo=10

and in the cleanUpArgs function I could just take each one out. What do you think @gziolo @aduth ?

@draganescu draganescu requested a review from gziolo March 7, 2019 10:33
Copy link
Member

@gziolo gziolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option would to not filter by prefix, but instead by command name. This way we would have shorter arguments for example:

npm run test-e2e:watch -- --interactive --slowmo=10

and in the cleanUpArgs function I could just take each one out.

Yes, that would work, too. I don't feel strongly about any of this.

packages/scripts/utils/cli.js Outdated Show resolved Hide resolved
@draganescu
Copy link
Contributor Author

@gziolo I have cleaned up the functions by moving the argument removal by prefix into the getCliArgs function of the process util. I've also rebased. We should be good to go :)

Copy link
Member

@gziolo gziolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one, we still need to document these new features. I will open a follow-up with more documentation improvements.

@gziolo gziolo merged commit 3cc3c89 into master Mar 15, 2019
@gziolo gziolo deleted the update/e2e-tests-options branch March 15, 2019 11:05
@youknowriad
Copy link
Contributor

Awesome work @draganescu

youknowriad pushed a commit that referenced this pull request Mar 20, 2019
* added an interactive argument to the test-e2e script

* mend

* added a way to bypass certain args from Jest, reverted to normal argument passing using double dash

* simplified the argument cleanup function

* simplified the argument cleanup function

* added a way to bypass certain args from Jest, reverted to normal argument passing using double dash

* simplified the argument cleanup function

* simplified the argument cleanup function

* cleaned up the code by moving the prefixed arguments removal to getcliargs
youknowriad pushed a commit that referenced this pull request Mar 20, 2019
* added an interactive argument to the test-e2e script

* mend

* added a way to bypass certain args from Jest, reverted to normal argument passing using double dash

* simplified the argument cleanup function

* simplified the argument cleanup function

* added a way to bypass certain args from Jest, reverted to normal argument passing using double dash

* simplified the argument cleanup function

* simplified the argument cleanup function

* cleaned up the code by moving the prefixed arguments removal to getcliargs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Enhancement A suggestion for improvement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants