[Feature] watch mode for e2e tests

[JoelEinbinder]

I'd really appreciate a watch mode. My hands hurt from running my test over and over.

---

If you are looking for the watch mode that runs tests on save, I found assigning the shortcut to Run Test at Cursor very handy:

Playwright VS Code Extension is likely to mitigate your request

VS Code extension makes writing and debugging tests convenient, there is no need to go between code and terminal back and forth.

[laurentpayot]

As a workaround with Playwright 1.12+ I’m using the test:e2e:watch npm script below to watch all my test files and run a single test file when it is modified (install chokidar-cli before):

{ "scripts": {
    "test:e2e": "playwright test --config=tests/playwright.config.ts",
    "test:e2e:watch": "chokidar 'tests/e2e/**/*.test.ts' -c 'npm run test:e2e -- --retries=0 {path}'",
}}

Of course modify these scripts according to your needs. Coupled to test.only it’s perfect for my usage.

[pavelfeldman]

This is going to be rebooted as a part of the debugging story. Closing with no replacement for now.

[iamyoki]

As a workaround with Playwright 1.12+ I’m using the test:e2e:watch npm script below to watch all my test files and run a single test file when it is modified (install chokidar-cli before):

{ "scripts": {
    "test:e2e": "playwright test --config=tests/playwright.config.ts",
    "test:e2e:watch": "chokidar 'tests/e2e/**/*.test.ts' -c 'npm run test:e2e -- --retries=0 {path}'",
}}

Of course modify these scripts according to your needs. Coupled to test.only it’s perfect for my usage.

This would be better :)
"test:e2e:watch": "chokidar e2e/**/*.{spec,test}.{js,ts} -c 'yarn run test:e2e'"

[iamyoki]

I made a library playwright-watch to achieve this.
See playwright-watch docs

playwright-watch

playwright + watch = palywright-watch

🙈🐵 Run Playwright in watch mode

Usage

use exactly the same as Playwright, only in watch mode

npx playwright-watch test

Note: There is a peer dependency on playwright.

Or you can install locally

yarn add playwright-watch -D

#or

npm install playwright-watch --save-dev

Add script to your package.json

{
  "test:watch": "playwright-watch test"
}

or

{
  "test": "playwright test",
  "test:watch": "playwright-watch exec yarn test"
}

CLI Details

run npx playwright-watch --help to see more details.

🙈 playwright-watch

🐵 Run Playwright in watch mode

Commands:
  playwright-watch <commands..>       Use the same as Playwright but under watch
                                      (e.g. playwright-watch test)     [default]
  playwright-watch exec <commands..>  Run commands when changes are detected
                                      (e.g. playwright-watch exec yarn test)

Options:
  -h, --help     Show help                                             [boolean]
  -v, --version  Show version number                                   [boolean]
  -c, --config   Configuration file of Playwright (default:
                 playwright.config.js), the `testMatch` prop will be watch path
                 (default: `.*(test|spec).(js|ts|mjs)`).

Examples:
  playwright-watch test
  playwright-watch test --config tests/playwright.config.js

  playwright-watch exec yarn playwright test
  playwright-watch exec yarn run test:e2e
  playwright-watch exec echo changed

Author

Yoki Yu (iamyoki)

[schickling]

Thanks for this tool @iamyoki as a workaround!

@pavelfeldman would be great if adding a watch mode could still be considered to be added to Playwright regardless. I’m planning on using PW as a general test runner (mostly for non-web tests), so this functionality would be quite important for my use cases. 🙏

[YonatanKra]

@pavelfeldman
We really need a watch mode that also leaves the browsers open.
I'd be happy to work on this if you could guide me.

[frederikhors]

@pavelfeldman We really need a watch mode that also leaves the browsers open. I'd be happy to work on this if you could guide me.

Yes please!

[dannyyassine-ce]

until then, another workaround using nodemon:

nodemon --watch playwright/specs --exec \"playwright test -c playwright.config.ts\"",

also works great when using with the inspector:

nodemon --watch playwright/specs --exec \"PWDEBUG=1 playwright test -c playwright.config.ts\"",

if using with typescript with custom tsconfig.json, I have set up a pw:dev script in my package.json to watch when tsc changes the output directory to restart playwright:

"pw:dev": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test\"",
"pw:dev:debug": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test:debug\"",
"pw:watch": "tsc --incremental -p playwright/specs/tsconfig.json --watch & tsc-alias -p playwright/specs/tsconfig.json -w",
"pw:test": "playwright test -c playwright.config.ts",
"pw:test:debug": "PWDEBUG=1 playwright test -c playwright.config.ts"

[iamyoki]

until then, another workaround using nodemon:

nodemon --watch playwright/specs --exec \"playwright test -c playwright.config.ts\"",

also works great when using with the inspector:

nodemon --watch playwright/specs --exec \"PWDEBUG=1 playwright test -c playwright.config.ts\"",

if using with typescript with custom tsconfig.json, I have set up a pw:dev script in my package.json to watch when tsc changes the output directory to restart playwright:

"pw:dev": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test\"",
"pw:dev:debug": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test:debug\"",
"pw:watch": "tsc --incremental -p playwright/specs/tsconfig.json --watch & tsc-alias -p playwright/specs/tsconfig.json -w",
"pw:test": "playwright test -c playwright.config.ts",
"pw:test:debug": "PWDEBUG=1 playwright test -c playwright.config.ts"

Before I decided to create playwright-watch, I have tried this way, the only problem is that if the save is triggered multiple times in a short period of time, playwright will concurrently run multiple processes and cause memory overflow.

[dannyyassine-ce]

until then, another workaround using nodemon:

nodemon --watch playwright/specs --exec \"playwright test -c playwright.config.ts\"",

also works great when using with the inspector:

nodemon --watch playwright/specs --exec \"PWDEBUG=1 playwright test -c playwright.config.ts\"",

if using with typescript with custom tsconfig.json, I have set up a pw:dev script in my package.json to watch when tsc changes the output directory to restart playwright:

"pw:dev": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test\"",
"pw:dev:debug": "yarn pw:watch & nodemon --watch playwright/tests-out --exec \"yarn pw:test:debug\"",
"pw:watch": "tsc --incremental -p playwright/specs/tsconfig.json --watch & tsc-alias -p playwright/specs/tsconfig.json -w",
"pw:test": "playwright test -c playwright.config.ts",
"pw:test:debug": "PWDEBUG=1 playwright test -c playwright.config.ts"

Before I decided to create playwright-watch, I have tried this way, the only problem is that if the save is triggered multiple times in a short period of time, playwright will concurrently run multiple processes and cause memory overflow.

you can always add a delay with changing files :) -> https://github.com/remy/nodemon#delaying-restarting

[cliffordfajardo]

We recently started using Playwright where I work & many peer teams are starting to move away from Cypress
I don't have much Cypress experience, I started off using Playwright

1 common thing I hear from former Cypress users is if there is something similar to the npm run cy:open command which opens the browser (non-headless) and watches for file changes
Simply adding as a data point ^

Resources:
If you have feel like Playwright is missing some features which you think are relavant to Cypress, please leave a comment here too so we can help the playwright ecosystem 🙏 : Github Discussion 11201 -- Playwright vs Cypress Feature Comparison - Share Questions & Experience

[iamzapata]

Hey, adding a bit to the discussion here. About 3 years ago I build a large e2e test suite with Cypress and I was in love with it. After a long time battling with cypress I came to also hate some parts of it and started looking for alternatives and found playwright.

This seems like a winner, but I just can't use it. It's impossible to be productive when writing new specs if I have to constantly type commands go back and forth between the test code and the test runner.

I haven't read all of the documentation yet, but is there something I'm missing? I think watch mode should be one of the things that should just work out of the box.

[timcash]

@pavelfeldman We really need a watch mode that also leaves the browsers open. I'd be happy to work on this if you could guide me.

This is needed, tests should be able to
1.) all run in the same instance of an open browser
2.) reload on test change
3.) reload on dependency change

[pavelfeldman]

@iamzapata I think you want this: https://github.com/microsoft/playwright-vscode

[franzvezuli]

A watch mode would be great, it's kind of the only deal-breaker between Playwright and Cypress in my opinion.

Is it this issue that's referred to in the docs?: https://playwright.dev/docs/test-components#planned-work

Planned work
Watch mode: watch mode is highly anticipated and should be relatively straightforward in implementation.

[lobsterkatie]

Ran into this trying to create a VSCode debugger profile for our playwright tests. (I know about https://github.com/microsoft/playwright-vscode (from this thread), but I just wanted to be able to tweak an existing debugger profile because it's faster than learning a whole new tool.)

I started by trying playwright-watch, but couldn't get it work. (The first problem was that our playwright.config.ts is in TS, but even once that was solved, it just hung forever and didn't actually run the test in question.) Eventually got it to work with nodemon. I'll share the results, in case it helps anyone else.

For context:

• We write in TS, run a monorepo using lerna and yarn workspaces, and use playwright to run our browser-based integration tests.

• The goal was to replicate the setup we have for our jest tests, wherein the test file currently open in the editor has its tests run in watch mode.

• The solution here requires a VSCode extension called Tasks Shell Input. That's just to handle the monorepo-i-ness of it all, though, so if that doesn't apply to you, then it (and the entirety of inputs below) can be skipped.

  (P.S. If anyone has a better idea of how to get an arbitrary segment of a path from within a launch.json configuration, I'm all ears, because I recognize this is pretty clunky. I just couldn't figure out any other way (besides the inputs entry below) to get the packageName part of a path like /some/path/here/<repo-name>/packages/<packageName>/test/some.test.ts using just the predefined variables here.)

launch.json

{
  "version": "0.2.0",
  "inputs": [
    // Get the name of the package containing the file in the active tab.
    {
      "id": "getPackageName",
      "type": "command",
      "command": "shellCommand.execute",
      "args": {
        // Get the current file's absolute path, chop off everything up to and including the repo's `packages`
        // directory, the split on `/` and take the first entry
        "command": "echo '${file}' | sed s/'.*sentry-javascript\\/packages\\/'// | grep --extended-regexp --only-matching --max-count 1 '[^\\/]+' | head -1",
        "cwd": "${workspaceFolder}",
        // normally `input` commands bring up a selector for the user, but given that there should only be one
        // choice here, this lets us skip the prompt
        "useSingleResult": true
      }
    }
  ],
  "configurations": [
    // Run a specific test file in watch mode (must have file in currently active tab when hitting the play button).
    // NOTE: If you try to run this and VSCode complains that the command `shellCommand.execute` can't be found, go
    // install the recommended extension Tasks Shell Input.
    {
      "name": "Debug playwright tests (just open file)",
      "type": "pwa-node",
      "cwd": "${workspaceFolder}/packages/${input:getPackageName}",
      "request": "launch",
      "runtimeExecutable": "yarn",
      "runtimeArgs": [
        // `nodemon` is basically `node --watch`
        "nodemon",
        // be default it only watches JS files, so have it watch TS files instead
        "--ext",
        "ts",
        "${workspaceFolder}/node_modules/playwright/node_modules/.bin/playwright",
        "test",
        "${file}"
      ],
      "skipFiles": ["<node_internals>/**"],
      "outFiles": ["${workspaceFolder}/**/*.js", "!**/node_modules/**"],
      "sourceMaps": true,
      "smartStep": true,
      "internalConsoleOptions": "openOnSessionStart",
      // show stdout and stderr output in the debug console
      "outputCapture": "std"
    }
  ]
}

[bichotll]

It would be nice if meanwhile the nodemon solution or similar were added to the documentation :)

[the-illarionov]

Dayumn, wanted to migrate to Playwright from Cypress so much (in LOVE with natural js async behavior), but absence of watch mode like in Cypress really stops me.

[zoltan-nz]

An awesome and super developer friendly solution to wrap playwright in a vitest runner (or jest, but it is soo 2022 ;) ). In this case managing, rerunning, debugging tests are much easier. Official example: https://github.com/vitest-dev/vitest/blob/main/examples/playwright/test/basic.test.ts

[alexbjorlig]

Correct me if I'm wrong, but if we use vitest like shown here - how do we then benefit from playwright fixtures extending the base for example?

[zoltan-nz]

@alexbjorlig You totally right, if you use different test runners, you get the third party test runner's benefits, but loose the playwright test runner underlying features. In terms of fixtures, we have to directly setup page fixtures, maybe in the beforeAll. Hopefully watch mode will be implemented in Playwright eventually. Playwright is such an amazing tool. Love it.

[SalahAdDin]

It is still in the plans, right?

[mbre777]

Watch mode is a must have feature in my opinion. Have been excited about Playwright and had plans to switch from Cypress, but not without having a watch mode.

[SalahAdDin]

Watch mode is a must-have feature in my opinion. Have been excited about Playwright and had plans to switch from Cypress, but not without having a watch mode.

Well, It is late for us, we switched to Playwright not knowing there was not watch mode.

[sand4rt]

Would be patient, i think they will add it sooner or later!👀

[MikeShi42]

Hey everyone! We’ve hacked together a visual watch mode for Playwright - it’ll run and pop open a debugger any time you hit save (similar to Cypress) - would love to hear what y’all think!

https://www.npmjs.com/package/@deploysentinel/playwright-watch

tl;dr docs:

Install:

npm i -D @deploysentinel/playwright-watch @deploysentinel/playwright

Run:

npx pw-watch --reporter @deploysentinel/playwright

[SalahAdDin]

Hey everyone! We’ve hacked together a visual watch mode for Playwright - it’ll run and pop open a debugger any time you hit save (similar to Cypress) - would love to hear what y’all think!

https://www.npmjs.com/package/@deploysentinel/playwright-watch

tl;dr docs:

Install:

npm i -D @deploysentinel/playwright-watch @deploysentinel/playwright

Run:

npx pw-watch --reporter @deploysentinel/playwright

Nice for checking, thanks!

[cliffordfajardo]

Hey everyone! We’ve hacked together a visual watch mode for Playwright - it’ll run and pop open a debugger any time you hit save (similar to Cypress) - would love to hear what y’all think!

tl;dr docs:

Install:

npm i -D @deploysentinel/playwright-watch @deploysentinel/playwright

Run:

npx pw-watch --reporter @deploysentinel/playwright

The developer experience and touches on looknice! 💙
Saw that the deploysentinel team made a blog post announcement with details 👀
https://www.deploysentinel.com/playwright

pw-debugger-sentintenel-demo.mp4

[RIP21]

Please implement this feature :)
My global-setup.ts takes easily 10-20 seconds (starts DB and Redis container, Backend, and Next.js Frontend) but then is fine to be reused as tests are isolated.
Restarting all this machinery over and over again takes a lot of time and having a watch mode that is not starting the global-setup.ts would be super nice.

I think I can work around it by starting it all on some unique predefined ports using some docker-compose before e2e starts. But that won't be as seamless and super reliable as having always absolutely clean test containers running on random ports. (I use testcontainers-node to start it all)

[aslushnikov]

This is now fixed with the new UI mode in Playwright 1.32.

• see https://github.com/microsoft/playwright/releases/tag/v1.32.0 for details
• video demo of the UI mode: https://www.youtube.com/watch?v=jF0yA-JLQW0

Please give UI mode a try and let us know (via filing new issues) if something doesn't work!

[RichiCoder1]

That's fantastic but...is there a way to watch without the UI? There's reasons launch a full UI just to watch tests and get feedback might not be ideal.

[aslushnikov]

@RichiCoder1 would you mind filing a separate issue and explaning your use case there? This way we'll be able to gauge interest and asses uses cases.

Thanks!

[RichiCoder1]

Created #21960 for headless watch mode. Lmk if I can phrase it better!

[dman777]

Just to mention, I do not use VS code (I use VIM) so if this feature is ever implemented please do not tightly couple it with VS code.

[luchillo17]

I'm wondering if there's a way to make the playwright E2E watch both the tests and the source code, I know it might not be a priority since the usual SDLC even with TDD is to have unit tests running on watch mode while we develop, and E2E gets added later at the end just to tie it all together, or to cover dark bugs regression testing.

It's just that part of the promise of the monorepo is to have dependency and colocation on other apps & libs in the monorepo, I was expecting the E2E to watch its dependencies just like apps & libs do theirs when serving or running tests.

[saneatsu]

If you want to watch in headless mode, try the following command.

ref: #21960 (comment)

$ PWTEST_WATCH=1 npx playwright test