-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PoC: Visual regression testing with Playground #62729
base: trunk
Are you sure you want to change the base?
Conversation
Size Change: 0 B Total Size: 1.76 MB ℹ️ View Unchanged
|
449a2d5
to
3c2989a
Compare
3c2989a
to
6fd5216
Compare
I tried to test this, and I'm not sure if I'm testing incorrectly or if there is something else going on. If I manually try the PR preview page with PR 59242, I receive an error...
|
You're testing correctly. The problem is the PR you entered (#59242), where the zipped artifact for the build from that PR is no longer available. This is because those artifacts have a maximum retention period of 90 days. I don't know the current setting for the retention days in Gutenberg, but that PR is already over 130 days old. @smithjw1, would you consider it a blocking limitation for the visual regression comparison if we couldn't test PRs older than 90 days? As a side note, the code that checks if the artifact is available is not working as expected, which is why the correct error is not being thrown: edit: The issue has been reported in WordPress/wordpress-playground#1568 |
No, not blocking. Automated testing shouldn't be an issue since the goal here is automated testing. I picked that PR because I know it has changes that the test should pick up. Thanks for logging that other issue! I'll look through more recent PRs to locate the change the test will reveal. |
Cool! Let me know if you need help designing a test for a specific regression. It would be nice to see how Playwright performs here. 🙌 |
b8f5a90
to
faad6a5
Compare
faad6a5
to
a3eefdf
Compare
Flaky tests detected in a3eefdf. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/9856148260
|
This is lovely @WunderBart, thank you so much for sharing! It's already great for running tests. I wonder if we could go a step further and record and replay that directly in the browser. I know Chrome devtools have this recorder: https://developer.chrome.com/docs/devtools/recorder I wonder if there are any ways of controlling it from a website or a browser extension. I know there are browser extensions that record to Playwright, I wonder if there's anything to run those recorded tests – even if they were restricted to a subset of commands, not the full power of Playwright. If that was possible, we could add fantastic debugging, error reporting, and E2E testing features to Playground. |
Warning: Type of PR label mismatch To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.
Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task. |
## Motivation for the change, related issues This is related to supporting visual regression testing of Gutenberg PRs with Playground. WordPress/gutenberg#62729 Importing content into a Playground makes testing easier. This will allow the supported `import-site` parameter to be used on this page as well. ## Implementation details This change adds a check for a the `import-site` query parameter and adds a step to the blueprint when it is found. The link has minimal validation, but I don't believe this creates a significant security concern. In theory someone could corrupt the blueprint by directing people to this URL with a malformed query parameter. Validating that the parameter is a URL, and not passing the query string or hash seemed to mitigate the most obvious offenses. ## Testing Instructions (or ideally a Blueprint) 1. Run locally 2. Visit `gutenberg.html` with a query parameter `import-site` pointing to a valid Playground ZIP import. 3. Confirm that Playground has imported content from ZIP file.
What?
This is a simple PoC for testing visual regressions using Playwright + Playground's Gutenberg PR Preview tool.
The example test compares the editor's default view with the one from a given PR number.
At this point, it does not use any tools from the Gutenberg E2E framework. It can, though, but to be functional, it doesn't really need anything from this repository apart from the Playwright dependency. It was just easier to put it here than create a new repo.
Warning
The Gutenberg PR Preview tool relies on the availability of build artifacts uploaded to GitHub. These artifacts have a maximum retention period of 90 days. Consequently, this PoC cannot test PRs older than 90 days.
How?
For example:
Headlessly:
Headfully:
Update snapshots:
cc: @adamziel @dmsnell @smithjw1 @griffbrad