[Feature] Support replaying Chrome Dev Tools Recorder panel's new JSON session recording format

[pirate]

Chrome added the Recorder panel to the Dev Tools in release v92, which provides an amazing UI for recording and editing puppeteer-style user flow scripts straight from the browser.

It used to only support exporting the user actions script as an autogenerated puppeteer JS script, but they recently added a more framework-agnostic JSON format in the v101 release that looks like this:

---

Feature Request

• 🙏 Playwright should add the ability to import and replay user scripts the new JSON format
• the docs here should be updated to show how to replay from JSON: https://playwright.dev/docs/next/network#record-and-replay-requests
• we should request the Chrome team to update their docs to show playwright can parse their format too: https://developer.chrome.com/docs/devtools/recorder/reference/#replay-with-external-libraries

---

Would the be useful to many people?

I think it will help massively reduce complexity in the ecosystem as tools will no longer need to provide a multitude of output format options to support exporting to puppeteer/pyppeteer/playwright/selenium/etc.. One common format to rule them all could be super helpful for everyone.

HAR/WARC/WACZ files plus these JSON files could then be shipped around as a complete unit needed to reproduce an entire interactive browsing session in a variety of replay tools. (I'd like to add this to to the internet archiving tool I maintain, ArchiveBox: ArchiveBox/ArchiveBox#51, and my friends over at browsertrix-crawler want it too: webrecorder/browsertrix-crawler#283)

---

JSON User Flow Format Example

Demo URL: https://jec.fyi/demo/recorder

Steps to reproduce in Chrome: View > Developer > Developer Tools > Recorder > Create a new recording > ⏺️ Start recording > ⏹️ End recording > ⬇️ Export > JSON

{
    "title": "Test Recording 4/11/2023 at 5:15:18 PM",
    "steps": [
        {
            "type": "setViewport",
            "width": 1869,
            "height": 264,
            "deviceScaleFactor": 1,
            "isMobile": false,
            "hasTouch": false,
            "isLandscape": false
        },
        {
            "type": "navigate",
            "assertedEvents": [
                {
                    "type": "navigation",
                    "url": "https://jec.fyi/demo/recorder",
                    "title": "Puppeteer recorder - test page"
                }
            ],
            "url": "https://jec.fyi/demo/recorder"
        },
        {
            "type": "click",
            "target": "main",
            "selectors": [
                [
                    "[data-testid='email']"
                ],
                [
                    "xpath///*[@data-testid=\"email\"]"
                ],
                [
                    "pierce/[data-testid='email']"
                ]
            ],
            "offsetX": 58.1776123046875,
            "offsetY": 8.579559326171875
        }
    ]
}

---

More Info

Release announcements for the Recorder pane and Export menu

• https://developer.chrome.com/blog/new-in-devtools-92/?utm_source=devtools#puppeteer-recorder
• https://developer.chrome.com/blog/new-in-devtools-101/#recorder

Reference docs about Recorder pane and the Export menu formats

• https://developer.chrome.com/docs/devtools/recorder/reference/
• https://developer.chrome.com/docs/devtools/recorder/reference/#recorder-extension
• https://developer.chrome.com/docs/devtools/recorder/reference/#export-flows
• https://developer.chrome.com/docs/devtools/recorder/reference/#replay-with-external-libraries
• https://developer.chrome.com/docs/devtools/recorder/extensions/
• https://developer.chrome.com/blog/extend-recorder/

[mxschmitt]

From our experience our recorder is much more powerful than third-party recorders since its optimised for Playwright. Do you miss functionality from our recorder? Then feel free to file a feature request for it! It also creates human readable Playwright scripts for all our languages out of the box. Since providing a separate recorder generator integration is outside of the scope for the Playwright project, I'm closing it for now.

If you are interested in writing a code generator for this Chrome file format, this can also be done as a third-party project.

[pirate]

I believe the intent of this JSON export feature is to create a tool-agnostic format that many replayers can share.

I think right now the entire browser automation ecosystem is burdened by having to translate each other's crowd-sourced python/js/etc. user scripts between languages and environments like puppeteer/selenium/playwright. The power of the tool-agnostic JSON format is that it's just the raw data needed, with none of the language-specific things required like imports, JS IIFE async setup, variable declaration lines, etc. that you'd have to translate.

I think Playwright is the best browser driver by far and is likely to win out, and I definitely appreciate the focus being on improving playwright's recording before implementing import/export stuff. From a bigger picture perspective it might still benefit the whole ecosystem to move away from code-generated imperative JS scripts and move towards a flat declarative data format. I see this as an olive branch from the Chromium team to start this process. (They could've released a proprietary format designed to encourage Chrome+puppeteer but it seems to be very flat and tool agnostic to encourage interop.)

I understand if playwright doesn't want to jump on board immediately, but I'd encourage revisiting this idea in 6 months if this new JSON takes off and ends up becoming a common interchange format for user flows.

[autonomobil]

It would enable non tech savy people to record "user-interactions" without setting up a coding environment, so really powerful to bring people onboard who else wouldn't have the chance to contribute.

So any plans in that regard?

I know there is:

• https://chrome.google.com/webstore/detail/playwright-chrome-recorde/bfnbgoehgplaehdceponclakmhlgjlpd
• https://chrome.google.com/webstore/detail/playwright-crx/jambeljnbnfbkcpnoiaedcabbgmnnlcd

but some random extension is a different story than official support

[pirate]

It's been a little while now @mxschmitt, any chance we could get ya'll to reconsider supporting a JSON/declarative user scripting interchange format?

I think it would bring together the contributor pools from many projects if we could all exchange user scrips in a standardized, tool-agnostic format. There's such a wide range of browser automation tools right now, but so many have created their own recorders + recording formats + user script repositories. We could get the pools of thousands of users and contributors across these projects sharing scripts, and it might even increase Playwright's adoption if people are no longer locked in to older Puppeteer/Selenium scripts as a sharing standard:

• https://katalon.com/web-testing
• https://www.mabl.com/cross-browser-testing
• https://www.selenium.dev/selenium-ide/
• https://www.browserstack.com/docs/automate/capabilities
• https://ghostinspector.com/features-non-coding/#test-recorder
• https://github.com/open-source-labs/spearmint
• and many more open-source tools and collections of puppeteer/selenium/playwright scripts...
  • https://github.com/zotero/translators
  • https://github.com/apify/crawlee
  • https://github.com/niespodd/browser-fingerprinting
  • https://github.com/checkly/puppeteer-examples
  • https://github.com/berstend/puppeteer-extra
  • https://github.com/BruceDone/awesome-crawler
  • https://github.com/transitive-bullshit/awesome-puppeteer
  • https://github.com/zhentaoo/puppeteer-deep
    (making this list of user script collections for my own benefit / others too, as it's useful to see how people are currently sharing these scripts in non-json formats)

I'd also be willing to personally contribute towards PRs to support this format if I know it's a possibility, as I know it takes real work and I don't want to just ask ya'll to implement this for free on your own.

Also to clarify: my intent for my own use in Archivebox would be to extend the JSON format with provisions for executing arbitrary JS, performing browser actions like taking a screenshot/PDF, etc. The JSON format just serves as a declarative base that can be extended by other tools without breaking cross-compatibility.

[daniel31x13]

This feature would have a huge positive impact on the whole web archiving industry.
I'd be happy to make use of it in Linkwarden as well.

[nullptrerror]

Just jumping in on the JSON/declarative script format discussion. Adding this to Playwright would be a game-changer. It'll let us swap scripts easily across different tools like Puppeteer and Selenium. Think about it - no more juggling different formats for each tool. We'd have a universal language for browser scripts!

This isn't just about making our lives easier (though it definitely will). It's about bringing together the browser automation community. More shared scripts mean more innovation and faster progress for everyone. Plus, it could make Playwright the go-to for people currently using other tools.

I'm up for helping out with this. I know it's a big ask, and I'm ready to pitch in with the work.

[surfmuggle]

Hello @mxschmitt you wrote

From our experience our recorder is much more powerful than third-party recorders since its optimised for Playwright.

In some companies installing a tool like playwright is quite a hassle (for example playwright is not available as a package in Microsoft Intune / Configuration Manager therefore the help desk must manually support the installation). External Contractors have to use Virtual Desktop Infrastructure (VDI). Some corporations refuse to install "exotic" software on a VDI.

The benefit of using the built in recorder (of msft edge and or google chrome) is that a these browsers are already available on most corporate computers.

Do you miss functionality from our recorder?

In the devtools recorder the recorded click path is displayed with screens and can be replayed.

If the playwright ui-mode GUI npx playwright test --ui could be combined with npx playwright codegen to see the commands (playwright actions) and the matching screen this would be great.