Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC: Visual regression testing with Playground #62729

Draft
wants to merge 3 commits into
base: trunk
Choose a base branch
from

Conversation

WunderBart
Copy link
Member

@WunderBart WunderBart commented Jun 21, 2024

What?

This is a simple PoC for testing visual regressions using Playwright + Playground's Gutenberg PR Preview tool.

The example test compares the editor's default view with the one from a given PR number.

At this point, it does not use any tools from the Gutenberg E2E framework. It can, though, but to be functional, it doesn't really need anything from this repository apart from the Playwright dependency. It was just easier to put it here than create a new repo.

Warning

The Gutenberg PR Preview tool relies on the availability of build artifacts uploaded to GitHub. These artifacts have a maximum retention period of 90 days. Consequently, this PoC cannot test PRs older than 90 days.

How?

PR_NUMBER=<pr-nr-to-compare> npm run test:visreg

For example:

  • Headlessly:

    PR_NUMBER=62633 npm run test:visreg
  • Headfully:

    PR_NUMBER=62633 npm run test:visreg -- --headed
  • Update snapshots:

    PR_NUMBER=62633 npm run test:visreg -- --update-snapshots

cc: @adamziel @dmsnell @smithjw1 @griffbrad

Copy link

github-actions bot commented Jun 21, 2024

Size Change: 0 B

Total Size: 1.76 MB

ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 951 B
build/annotations/index.min.js 2.26 kB
build/api-fetch/index.min.js 2.31 kB
build/autop/index.min.js 2.12 kB
build/blob/index.min.js 579 B
build/block-directory/index.min.js 7.31 kB
build/block-directory/style-rtl.css 1.01 kB
build/block-directory/style.css 1.01 kB
build/block-editor/content-rtl.css 4.58 kB
build/block-editor/content.css 4.58 kB
build/block-editor/default-editor-styles-rtl.css 394 B
build/block-editor/default-editor-styles.css 394 B
build/block-editor/index.min.js 254 kB
build/block-editor/style-rtl.css 16 kB
build/block-editor/style.css 16 kB
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 90 B
build/block-library/blocks/archives/style.css 90 B
build/block-library/blocks/audio/editor-rtl.css 149 B
build/block-library/blocks/audio/editor.css 151 B
build/block-library/blocks/audio/style-rtl.css 132 B
build/block-library/blocks/audio/style.css 132 B
build/block-library/blocks/audio/theme-rtl.css 134 B
build/block-library/blocks/audio/theme.css 134 B
build/block-library/blocks/avatar/editor-rtl.css 115 B
build/block-library/blocks/avatar/editor.css 115 B
build/block-library/blocks/avatar/style-rtl.css 104 B
build/block-library/blocks/avatar/style.css 104 B
build/block-library/blocks/button/editor-rtl.css 310 B
build/block-library/blocks/button/editor.css 310 B
build/block-library/blocks/button/style-rtl.css 538 B
build/block-library/blocks/button/style.css 538 B
build/block-library/blocks/buttons/editor-rtl.css 336 B
build/block-library/blocks/buttons/editor.css 336 B
build/block-library/blocks/buttons/style-rtl.css 328 B
build/block-library/blocks/buttons/style.css 328 B
build/block-library/blocks/calendar/style-rtl.css 240 B
build/block-library/blocks/calendar/style.css 240 B
build/block-library/blocks/categories/editor-rtl.css 113 B
build/block-library/blocks/categories/editor.css 112 B
build/block-library/blocks/categories/style-rtl.css 124 B
build/block-library/blocks/categories/style.css 124 B
build/block-library/blocks/code/editor-rtl.css 53 B
build/block-library/blocks/code/editor.css 53 B
build/block-library/blocks/code/style-rtl.css 121 B
build/block-library/blocks/code/style.css 121 B
build/block-library/blocks/code/theme-rtl.css 122 B
build/block-library/blocks/code/theme.css 122 B
build/block-library/blocks/columns/editor-rtl.css 108 B
build/block-library/blocks/columns/editor.css 108 B
build/block-library/blocks/columns/style-rtl.css 420 B
build/block-library/blocks/columns/style.css 420 B
build/block-library/blocks/comment-author-avatar/editor-rtl.css 124 B
build/block-library/blocks/comment-author-avatar/editor.css 124 B
build/block-library/blocks/comment-content/style-rtl.css 90 B
build/block-library/blocks/comment-content/style.css 90 B
build/block-library/blocks/comment-template/style-rtl.css 200 B
build/block-library/blocks/comment-template/style.css 199 B
build/block-library/blocks/comments-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/comments-pagination-numbers/editor.css 121 B
build/block-library/blocks/comments-pagination/editor-rtl.css 221 B
build/block-library/blocks/comments-pagination/editor.css 211 B
build/block-library/blocks/comments-pagination/style-rtl.css 234 B
build/block-library/blocks/comments-pagination/style.css 231 B
build/block-library/blocks/comments-title/editor-rtl.css 75 B
build/block-library/blocks/comments-title/editor.css 75 B
build/block-library/blocks/comments/editor-rtl.css 832 B
build/block-library/blocks/comments/editor.css 832 B
build/block-library/blocks/comments/style-rtl.css 632 B
build/block-library/blocks/comments/style.css 631 B
build/block-library/blocks/cover/editor-rtl.css 668 B
build/block-library/blocks/cover/editor.css 669 B
build/block-library/blocks/cover/style-rtl.css 1.62 kB
build/block-library/blocks/cover/style.css 1.6 kB
build/block-library/blocks/details/editor-rtl.css 65 B
build/block-library/blocks/details/editor.css 65 B
build/block-library/blocks/details/style-rtl.css 86 B
build/block-library/blocks/details/style.css 86 B
build/block-library/blocks/embed/editor-rtl.css 314 B
build/block-library/blocks/embed/editor.css 314 B
build/block-library/blocks/embed/style-rtl.css 419 B
build/block-library/blocks/embed/style.css 419 B
build/block-library/blocks/embed/theme-rtl.css 133 B
build/block-library/blocks/embed/theme.css 133 B
build/block-library/blocks/file/editor-rtl.css 326 B
build/block-library/blocks/file/editor.css 326 B
build/block-library/blocks/file/style-rtl.css 278 B
build/block-library/blocks/file/style.css 279 B
build/block-library/blocks/file/view.min.js 324 B
build/block-library/blocks/footnotes/style-rtl.css 198 B
build/block-library/blocks/footnotes/style.css 197 B
build/block-library/blocks/form-input/editor-rtl.css 229 B
build/block-library/blocks/form-input/editor.css 229 B
build/block-library/blocks/form-input/style-rtl.css 342 B
build/block-library/blocks/form-input/style.css 342 B
build/block-library/blocks/form-submission-notification/editor-rtl.css 344 B
build/block-library/blocks/form-submission-notification/editor.css 341 B
build/block-library/blocks/form-submit-button/style-rtl.css 69 B
build/block-library/blocks/form-submit-button/style.css 69 B
build/block-library/blocks/form/view.min.js 470 B
build/block-library/blocks/freeform/editor-rtl.css 2.6 kB
build/block-library/blocks/freeform/editor.css 2.6 kB
build/block-library/blocks/gallery/editor-rtl.css 958 B
build/block-library/blocks/gallery/editor.css 962 B
build/block-library/blocks/gallery/style-rtl.css 1.71 kB
build/block-library/blocks/gallery/style.css 1.71 kB
build/block-library/blocks/gallery/theme-rtl.css 108 B
build/block-library/blocks/gallery/theme.css 108 B
build/block-library/blocks/group/editor-rtl.css 402 B
build/block-library/blocks/group/editor.css 402 B
build/block-library/blocks/group/style-rtl.css 103 B
build/block-library/blocks/group/style.css 103 B
build/block-library/blocks/group/theme-rtl.css 79 B
build/block-library/blocks/group/theme.css 79 B
build/block-library/blocks/heading/style-rtl.css 188 B
build/block-library/blocks/heading/style.css 188 B
build/block-library/blocks/html/editor-rtl.css 346 B
build/block-library/blocks/html/editor.css 347 B
build/block-library/blocks/image/editor-rtl.css 845 B
build/block-library/blocks/image/editor.css 843 B
build/block-library/blocks/image/style-rtl.css 1.54 kB
build/block-library/blocks/image/style.css 1.54 kB
build/block-library/blocks/image/theme-rtl.css 137 B
build/block-library/blocks/image/theme.css 137 B
build/block-library/blocks/image/view.min.js 1.54 kB
build/block-library/blocks/latest-comments/style-rtl.css 355 B
build/block-library/blocks/latest-comments/style.css 354 B
build/block-library/blocks/latest-posts/editor-rtl.css 204 B
build/block-library/blocks/latest-posts/editor.css 204 B
build/block-library/blocks/latest-posts/style-rtl.css 509 B
build/block-library/blocks/latest-posts/style.css 510 B
build/block-library/blocks/list/style-rtl.css 104 B
build/block-library/blocks/list/style.css 104 B
build/block-library/blocks/media-text/editor-rtl.css 304 B
build/block-library/blocks/media-text/editor.css 303 B
build/block-library/blocks/media-text/style-rtl.css 516 B
build/block-library/blocks/media-text/style.css 515 B
build/block-library/blocks/more/editor-rtl.css 427 B
build/block-library/blocks/more/editor.css 427 B
build/block-library/blocks/navigation-link/editor-rtl.css 663 B
build/block-library/blocks/navigation-link/editor.css 664 B
build/block-library/blocks/navigation-link/style-rtl.css 192 B
build/block-library/blocks/navigation-link/style.css 191 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 295 B
build/block-library/blocks/navigation-submenu/editor.css 294 B
build/block-library/blocks/navigation/editor-rtl.css 2.2 kB
build/block-library/blocks/navigation/editor.css 2.21 kB
build/block-library/blocks/navigation/style-rtl.css 2.26 kB
build/block-library/blocks/navigation/style.css 2.25 kB
build/block-library/blocks/navigation/view.min.js 1.03 kB
build/block-library/blocks/nextpage/editor-rtl.css 392 B
build/block-library/blocks/nextpage/editor.css 392 B
build/block-library/blocks/page-list/editor-rtl.css 378 B
build/block-library/blocks/page-list/editor.css 378 B
build/block-library/blocks/page-list/style-rtl.css 175 B
build/block-library/blocks/page-list/style.css 175 B
build/block-library/blocks/paragraph/editor-rtl.css 236 B
build/block-library/blocks/paragraph/editor.css 236 B
build/block-library/blocks/paragraph/style-rtl.css 341 B
build/block-library/blocks/paragraph/style.css 340 B
build/block-library/blocks/post-author/style-rtl.css 175 B
build/block-library/blocks/post-author/style.css 176 B
build/block-library/blocks/post-comments-form/editor-rtl.css 96 B
build/block-library/blocks/post-comments-form/editor.css 96 B
build/block-library/blocks/post-comments-form/style-rtl.css 522 B
build/block-library/blocks/post-comments-form/style.css 522 B
build/block-library/blocks/post-content/editor-rtl.css 74 B
build/block-library/blocks/post-content/editor.css 74 B
build/block-library/blocks/post-date/style-rtl.css 62 B
build/block-library/blocks/post-date/style.css 62 B
build/block-library/blocks/post-excerpt/editor-rtl.css 71 B
build/block-library/blocks/post-excerpt/editor.css 71 B
build/block-library/blocks/post-excerpt/style-rtl.css 141 B
build/block-library/blocks/post-excerpt/style.css 141 B
build/block-library/blocks/post-featured-image/editor-rtl.css 729 B
build/block-library/blocks/post-featured-image/editor.css 726 B
build/block-library/blocks/post-featured-image/style-rtl.css 341 B
build/block-library/blocks/post-featured-image/style.css 341 B
build/block-library/blocks/post-navigation-link/style-rtl.css 215 B
build/block-library/blocks/post-navigation-link/style.css 214 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 399 B
build/block-library/blocks/post-template/style.css 398 B
build/block-library/blocks/post-terms/style-rtl.css 96 B
build/block-library/blocks/post-terms/style.css 96 B
build/block-library/blocks/post-time-to-read/style-rtl.css 70 B
build/block-library/blocks/post-time-to-read/style.css 70 B
build/block-library/blocks/post-title/style-rtl.css 100 B
build/block-library/blocks/post-title/style.css 100 B
build/block-library/blocks/preformatted/style-rtl.css 125 B
build/block-library/blocks/preformatted/style.css 125 B
build/block-library/blocks/pullquote/editor-rtl.css 134 B
build/block-library/blocks/pullquote/editor.css 134 B
build/block-library/blocks/pullquote/style-rtl.css 342 B
build/block-library/blocks/pullquote/style.css 342 B
build/block-library/blocks/pullquote/theme-rtl.css 167 B
build/block-library/blocks/pullquote/theme.css 167 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 121 B
build/block-library/blocks/query-pagination-numbers/editor.css 118 B
build/block-library/blocks/query-pagination/editor-rtl.css 220 B
build/block-library/blocks/query-pagination/editor.css 208 B
build/block-library/blocks/query-pagination/style-rtl.css 287 B
build/block-library/blocks/query-pagination/style.css 283 B
build/block-library/blocks/query-title/style-rtl.css 64 B
build/block-library/blocks/query-title/style.css 64 B
build/block-library/blocks/query/editor-rtl.css 514 B
build/block-library/blocks/query/editor.css 513 B
build/block-library/blocks/query/view.min.js 958 B
build/block-library/blocks/quote/style-rtl.css 238 B
build/block-library/blocks/quote/style.css 238 B
build/block-library/blocks/quote/theme-rtl.css 221 B
build/block-library/blocks/quote/theme.css 225 B
build/block-library/blocks/read-more/style-rtl.css 138 B
build/block-library/blocks/read-more/style.css 138 B
build/block-library/blocks/rss/editor-rtl.css 101 B
build/block-library/blocks/rss/editor.css 101 B
build/block-library/blocks/rss/style-rtl.css 288 B
build/block-library/blocks/rss/style.css 287 B
build/block-library/blocks/search/editor-rtl.css 183 B
build/block-library/blocks/search/editor.css 183 B
build/block-library/blocks/search/style-rtl.css 672 B
build/block-library/blocks/search/style.css 671 B
build/block-library/blocks/search/theme-rtl.css 113 B
build/block-library/blocks/search/theme.css 113 B
build/block-library/blocks/search/view.min.js 475 B
build/block-library/blocks/separator/editor-rtl.css 100 B
build/block-library/blocks/separator/editor.css 100 B
build/block-library/blocks/separator/style-rtl.css 248 B
build/block-library/blocks/separator/style.css 248 B
build/block-library/blocks/separator/theme-rtl.css 195 B
build/block-library/blocks/separator/theme.css 195 B
build/block-library/blocks/shortcode/editor-rtl.css 286 B
build/block-library/blocks/shortcode/editor.css 286 B
build/block-library/blocks/site-logo/editor-rtl.css 806 B
build/block-library/blocks/site-logo/editor.css 803 B
build/block-library/blocks/site-logo/style-rtl.css 218 B
build/block-library/blocks/site-logo/style.css 218 B
build/block-library/blocks/site-tagline/editor-rtl.css 87 B
build/block-library/blocks/site-tagline/editor.css 87 B
build/block-library/blocks/site-title/editor-rtl.css 123 B
build/block-library/blocks/site-title/editor.css 123 B
build/block-library/blocks/site-title/style-rtl.css 71 B
build/block-library/blocks/site-title/style.css 71 B
build/block-library/blocks/social-link/editor-rtl.css 338 B
build/block-library/blocks/social-link/editor.css 338 B
build/block-library/blocks/social-links/editor-rtl.css 676 B
build/block-library/blocks/social-links/editor.css 675 B
build/block-library/blocks/social-links/style-rtl.css 1.51 kB
build/block-library/blocks/social-links/style.css 1.5 kB
build/block-library/blocks/spacer/editor-rtl.css 346 B
build/block-library/blocks/spacer/editor.css 346 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 394 B
build/block-library/blocks/table/editor.css 394 B
build/block-library/blocks/table/style-rtl.css 640 B
build/block-library/blocks/table/style.css 639 B
build/block-library/blocks/table/theme-rtl.css 152 B
build/block-library/blocks/table/theme.css 152 B
build/block-library/blocks/tag-cloud/style-rtl.css 266 B
build/block-library/blocks/tag-cloud/style.css 265 B
build/block-library/blocks/template-part/editor-rtl.css 393 B
build/block-library/blocks/template-part/editor.css 393 B
build/block-library/blocks/template-part/theme-rtl.css 113 B
build/block-library/blocks/template-part/theme.css 113 B
build/block-library/blocks/term-description/style-rtl.css 108 B
build/block-library/blocks/term-description/style.css 108 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 165 B
build/block-library/blocks/text-columns/style.css 165 B
build/block-library/blocks/verse/style-rtl.css 98 B
build/block-library/blocks/verse/style.css 98 B
build/block-library/blocks/video/editor-rtl.css 553 B
build/block-library/blocks/video/editor.css 554 B
build/block-library/blocks/video/style-rtl.css 192 B
build/block-library/blocks/video/style.css 192 B
build/block-library/blocks/video/theme-rtl.css 134 B
build/block-library/blocks/video/theme.css 134 B
build/block-library/classic-rtl.css 179 B
build/block-library/classic.css 179 B
build/block-library/common-rtl.css 1.1 kB
build/block-library/common.css 1.1 kB
build/block-library/editor-elements-rtl.css 75 B
build/block-library/editor-elements.css 75 B
build/block-library/editor-rtl.css 11.9 kB
build/block-library/editor.css 11.9 kB
build/block-library/elements-rtl.css 54 B
build/block-library/elements.css 54 B
build/block-library/index.min.js 219 kB
build/block-library/reset-rtl.css 472 B
build/block-library/reset.css 472 B
build/block-library/style-rtl.css 14.6 kB
build/block-library/style.css 14.6 kB
build/block-library/theme-rtl.css 702 B
build/block-library/theme.css 707 B
build/block-serialization-default-parser/index.min.js 1.12 kB
build/block-serialization-spec-parser/index.min.js 2.87 kB
build/blocks/index.min.js 52.3 kB
build/commands/index.min.js 15.2 kB
build/commands/style-rtl.css 955 B
build/commands/style.css 952 B
build/components/index.min.js 228 kB
build/components/style-rtl.css 12.1 kB
build/components/style.css 12.1 kB
build/compose/index.min.js 12.9 kB
build/core-commands/index.min.js 2.77 kB
build/core-data/index.min.js 72.6 kB
build/customize-widgets/index.min.js 10.9 kB
build/customize-widgets/style-rtl.css 1.35 kB
build/customize-widgets/style.css 1.35 kB
build/data-controls/index.min.js 641 B
build/data/index.min.js 8.98 kB
build/date/index.min.js 18 kB
build/deprecated/index.min.js 458 B
build/dom-ready/index.min.js 325 B
build/dom/index.min.js 4.65 kB
build/edit-post/classic-rtl.css 578 B
build/edit-post/classic.css 580 B
build/edit-post/index.min.js 12.5 kB
build/edit-post/style-rtl.css 2.34 kB
build/edit-post/style.css 2.33 kB
build/edit-site/index.min.js 209 kB
build/edit-site/posts-rtl.css 6.54 kB
build/edit-site/posts.css 6.54 kB
build/edit-site/style-rtl.css 11.6 kB
build/edit-site/style.css 11.6 kB
build/edit-widgets/index.min.js 17.6 kB
build/edit-widgets/style-rtl.css 4.18 kB
build/edit-widgets/style.css 4.18 kB
build/editor/index.min.js 98 kB
build/editor/style-rtl.css 9.1 kB
build/editor/style.css 9.1 kB
build/element/index.min.js 4.83 kB
build/escape-html/index.min.js 537 B
build/format-library/index.min.js 8.07 kB
build/format-library/style-rtl.css 494 B
build/format-library/style.css 493 B
build/hooks/index.min.js 1.54 kB
build/html-entities/index.min.js 445 B
build/i18n/index.min.js 3.58 kB
build/interactivity/debug.min.js 16.5 kB
build/interactivity/file.min.js 447 B
build/interactivity/image.min.js 1.68 kB
build/interactivity/index.min.js 13.4 kB
build/interactivity/navigation.min.js 1.16 kB
build/interactivity/query.min.js 742 B
build/interactivity/router.min.js 2.8 kB
build/interactivity/search.min.js 615 B
build/is-shallow-equal/index.min.js 526 B
build/keyboard-shortcuts/index.min.js 1.31 kB
build/keycodes/index.min.js 1.46 kB
build/list-reusable-blocks/index.min.js 2.16 kB
build/list-reusable-blocks/style-rtl.css 846 B
build/list-reusable-blocks/style.css 846 B
build/media-utils/index.min.js 2.92 kB
build/modules/importmap-polyfill.min.js 12.3 kB
build/notices/index.min.js 946 B
build/nux/index.min.js 1.58 kB
build/nux/style-rtl.css 749 B
build/nux/style.css 745 B
build/patterns/index.min.js 7.35 kB
build/patterns/style-rtl.css 687 B
build/patterns/style.css 685 B
build/plugins/index.min.js 1.81 kB
build/preferences-persistence/index.min.js 2.06 kB
build/preferences/index.min.js 2.9 kB
build/preferences/style-rtl.css 578 B
build/preferences/style.css 578 B
build/primitives/index.min.js 829 B
build/priority-queue/index.min.js 1.54 kB
build/private-apis/index.min.js 1.01 kB
build/react-i18n/index.min.js 630 B
build/react-refresh-entry/index.min.js 9.47 kB
build/react-refresh-runtime/index.min.js 6.76 kB
build/redux-routine/index.min.js 2.69 kB
build/reusable-blocks/index.min.js 2.73 kB
build/reusable-blocks/style-rtl.css 256 B
build/reusable-blocks/style.css 256 B
build/rich-text/index.min.js 10.1 kB
build/router/index.min.js 1.96 kB
build/server-side-render/index.min.js 1.94 kB
build/shortcode/index.min.js 1.4 kB
build/style-engine/index.min.js 2.01 kB
build/token-list/index.min.js 581 B
build/url/index.min.js 3.85 kB
build/vendors/react-dom.min.js 42.8 kB
build/vendors/react-jsx-runtime.min.js 560 B
build/vendors/react.min.js 2.65 kB
build/viewport/index.min.js 965 B
build/warning/index.min.js 250 B
build/widgets/index.min.js 7.19 kB
build/widgets/style-rtl.css 1.16 kB
build/widgets/style.css 1.16 kB
build/wordcount/index.min.js 1.03 kB

compressed-size-action

@WunderBart WunderBart changed the title Create an example visual regression test using Playground PoC: Visual regression testing with Playground Jun 21, 2024
@smithjw1
Copy link

smithjw1 commented Jul 1, 2024

I tried to test this, and I'm not sure if I'm testing incorrectly or if there is something else going on.

If I manually try the PR preview page with PR 59242, I receive an error...

/wp-admin/
PHP 8.3 - WP 6.5 - ⚠️ Storage: None



Error
Oops! There was a problem starting Playground. To figure out what went wrong, please take a look at the error logs provided below. If you see an "Invalid blueprint error," the logs will point out the specific step causing the issue. You can then double-check your blueprint. For more help, you can also [visit our documentation.](https://wordpress.github.io/wordpress-playground/blueprints-api/troubleshoot-and-debug-blueprints)

Don't show again
Search logs
[01-Jul-2024 21:04:35 UTC] PHP Deprecated: Hook http_api_transports is deprecated since version 6.4.0 with no alternative available. in /wordpress/wp-includes/functions.php on line 135
[01-Jul-2024 21:04:35 UTC] PHP Deprecated: Hook http_api_transports is deprecated since version 6.4.0 with no alternative available. in /wordpress/wp-includes/functions.php on line 135
[01-Jul-2024 21:04:35 UTC] JavaScript Error: Error when executing the blueprint step #2 ({"step":"writeFile","path":"/wordpress/pr/pr.zip","data":{"resource":"url","url":"/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242","caption":"Downloading Gutenberg PR 59242"},"progress":{"weight":2,"caption":"Applying Gutenberg PR 59242"}}) : Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242".
Check if the URL is correct and the server is reachable.
If it is reachable, the server might be blocking the request.
Check the browser console and network tabs for more information.

## Does the console show the error "No 'Access-Control-Allow-Origin' header"?

This means the server that hosts your file does not allow requests from other sites
(cross-origin requests, or CORS).You need to move the asset to a server that allows
cross-origin file downloads. Learn more about CORS at
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

If your file is on GitHub, load it from "raw.githubusercontent.com".
Here's how to do that:

1. Start with the original GitHub URL of the file. For example:
https://github.com/username/repository/blob/branch/filename.
2. Replace "github.com" with "raw.githubusercontent.com".
3. Remove the "/blob/" part of the URL.

The resulting URL should look like this:
https://raw.githubusercontent.com/username/repository/branch/filename

Error:
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242"
Error: Error when executing the blueprint step #2 ({"step":"writeFile","path":"/wordpress/pr/pr.zip","data":{"resource":"url","url":"/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242","caption":"Downloading Gutenberg PR 59242"},"progress":{"weight":2,"caption":"Applying Gutenberg PR 59242"}}) : Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242".
Check if the URL is correct and the server is reachable.
If it is reachable, the server might be blocking the request.
Check the browser console and network tabs for more information.

## Does the console show the error "No 'Access-Control-Allow-Origin' header"?

This means the server that hosts your file does not allow requests from other sites
(cross-origin requests, or CORS).You need to move the asset to a server that allows
cross-origin file downloads. Learn more about CORS at
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

If your file is on GitHub, load it from "raw.githubusercontent.com".
Here's how to do that:

1. Start with the original GitHub URL of the file. For example:
https://github.com/username/repository/blob/branch/filename.
2. Replace "github.com" with "raw.githubusercontent.com".
3. Remove the "/blob/" part of the URL.

The resulting URL should look like this:
https://raw.githubusercontent.com/username/repository/branch/filename

Error:
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242"
at Object.run (https://playground.wordpress.net/assets/config-1553725b.js:645:14900)
at async Cd (https://playground.wordpress.net/assets/config-1553725b.js:645:16582)
at async Fd (https://playground.wordpress.net/assets/config-1553725b.js:645:17215)
[01-Jul-2024 21:04:35 UTC] JavaScript Error: Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242".
Check if the URL is correct and the server is reachable.
If it is reachable, the server might be blocking the request.
Check the browser console and network tabs for more information.

## Does the console show the error "No 'Access-Control-Allow-Origin' header"?

This means the server that hosts your file does not allow requests from other sites
(cross-origin requests, or CORS).You need to move the asset to a server that allows
cross-origin file downloads. Learn more about CORS at
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

If your file is on GitHub, load it from "raw.githubusercontent.com".
Here's how to do that:

1. Start with the original GitHub URL of the file. For example:
https://github.com/username/repository/blob/branch/filename.
2. Replace "github.com" with "raw.githubusercontent.com".
3. Remove the "/blob/" part of the URL.

The resulting URL should look like this:
https://raw.githubusercontent.com/username/repository/branch/filename

Error:
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242"
at $a.resolve (https://playground.wordpress.net/assets/config-1553725b.js:608:5344)
[01-Jul-2024 21:04:35 UTC] JavaScript Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242".
Check if the URL is correct and the server is reachable.
If it is reachable, the server might be blocking the request.
Check the browser console and network tabs for more information.

## Does the console show the error "No 'Access-Control-Allow-Origin' header"?

This means the server that hosts your file does not allow requests from other sites
(cross-origin requests, or CORS).You need to move the asset to a server that allows
cross-origin file downloads. Learn more about CORS at
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

If your file is on GitHub, load it from "raw.githubusercontent.com".
Here's how to do that:

1. Start with the original GitHub URL of the file. For example:
https://github.com/username/repository/blob/branch/filename.
2. Replace "github.com" with "raw.githubusercontent.com".
3. Remove the "/blob/" part of the URL.

The resulting URL should look like this:
https://raw.githubusercontent.com/username/repository/branch/filename

Error:
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242"
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242".
Check if the URL is correct and the server is reachable.
If it is reachable, the server might be blocking the request.
Check the browser console and network tabs for more information.

## Does the console show the error "No 'Access-Control-Allow-Origin' header"?

This means the server that hosts your file does not allow requests from other sites
(cross-origin requests, or CORS).You need to move the asset to a server that allows
cross-origin file downloads. Learn more about CORS at
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

If your file is on GitHub, load it from "raw.githubusercontent.com".
Here's how to do that:

1. Start with the original GitHub URL of the file. For example:
https://github.com/username/repository/blob/branch/filename.
2. Replace "github.com" with "raw.githubusercontent.com".
3. Remove the "/blob/" part of the URL.

The resulting URL should look like this:
https://raw.githubusercontent.com/username/repository/branch/filename

Error:
Error: Could not download "/plugin-proxy.php?org=WordPress&repo=gutenberg&workflow=Build%20Gutenberg%20Plugin%20Zip&artifact=gutenberg-plugin&pr=59242"
at $a.resolve (https://playground.wordpress.net/assets/config-1553725b.js:608:5344)

@WunderBart
Copy link
Member Author

WunderBart commented Jul 2, 2024

I tried to test this, and I'm not sure if I'm testing incorrectly or if there is something else going on.

You're testing correctly. The problem is the PR you entered (#59242), where the zipped artifact for the build from that PR is no longer available. This is because those artifacts have a maximum retention period of 90 days. I don't know the current setting for the retention days in Gutenberg, but that PR is already over 130 days old.

@smithjw1, would you consider it a blocking limitation for the visual regression comparison if we couldn't test PRs older than 90 days?

As a side note, the code that checks if the artifact is available is not working as expected, which is why the correct error is not being thrown: The PR 59242 does not exist or GitHub CI did not finish building it yet. The response is always 200, regardless of the availability. The only difference is that if the artifact is unavailable, the payload will be { "error": "Request failed" }. I'm going to make an issue for that. cc: @adamziel

edit:

The issue has been reported in WordPress/wordpress-playground#1568

@smithjw1
Copy link

smithjw1 commented Jul 2, 2024

would you consider it a blocking limitation for the visual regression comparison if we couldn't test PRs older than 90 days?

No, not blocking. Automated testing shouldn't be an issue since the goal here is automated testing. I picked that PR because I know it has changes that the test should pick up.

Thanks for logging that other issue! I'll look through more recent PRs to locate the change the test will reveal.

@WunderBart
Copy link
Member Author

Thanks for logging that other issue! I'll look through more recent PRs to locate the change the test will reveal.

Cool! Let me know if you need help designing a test for a specific regression. It would be nice to see how Playwright performs here. 🙌

Copy link

github-actions bot commented Jul 9, 2024

Flaky tests detected in a3eefdf.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/9856148260
📝 Reported issues:

@adamziel
Copy link
Contributor

adamziel commented Jul 9, 2024

This is lovely @WunderBart, thank you so much for sharing!

It's already great for running tests. I wonder if we could go a step further and record and replay that directly in the browser. I know Chrome devtools have this recorder:

https://developer.chrome.com/docs/devtools/recorder

I wonder if there are any ways of controlling it from a website or a browser extension. I know there are browser extensions that record to Playwright, I wonder if there's anything to run those recorded tests – even if they were restricted to a subset of commands, not the full power of Playwright. If that was possible, we could add fantastic debugging, error reporting, and E2E testing features to Playground.

Copy link

github-actions bot commented Jul 9, 2024

Warning: Type of PR label mismatch

To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.

  • Type-related labels to choose from: [Type] Automated Testing, [Type] Breaking Change, [Type] Bug, [Type] Build Tooling, [Type] Code Quality, [Type] Copy, [Type] Developer Documentation, [Type] Enhancement, [Type] Experimental, [Type] Feature, [Type] New API, [Type] Task, [Type] Technical Prototype, [Type] Performance, [Type] Project Management, [Type] Regression, [Type] Security, [Type] WP Core Ticket, Backport from WordPress Core.
  • Labels found: Needs Author's Reply.

Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task.

bgrgicak pushed a commit to WordPress/wordpress-playground that referenced this pull request Jul 16, 2024
## Motivation for the change, related issues

This is related to supporting visual regression testing of Gutenberg PRs
with Playground.
WordPress/gutenberg#62729

Importing content into a Playground makes testing easier. This will
allow the supported `import-site` parameter to be used on this page as
well.

## Implementation details

This change adds a check for a the `import-site` query parameter and
adds a step to the blueprint when it is found.

The link has minimal validation, but I don't believe this creates a
significant security concern. In theory someone could corrupt the
blueprint by directing people to this URL with a malformed query
parameter.

Validating that the parameter is a URL, and not passing the query string
or hash seemed to mitigate the most obvious offenses.

## Testing Instructions (or ideally a Blueprint)

 1. Run locally 
2. Visit `gutenberg.html` with a query parameter `import-site` pointing
to a valid Playground ZIP import.
 3. Confirm that Playground has imported content from ZIP file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Needs Author's Reply
Development

Successfully merging this pull request may close these issues.

3 participants