HTML API: Use full parser for html5lib tests #7117

sirreal · 2024-07-31T17:32:46Z

Previously, the html5lib tests have only run in the fragment parser mode, but in #6977 a full parser was introduced to the HTML API. In this patch, the html5lib tests are running in the full parser mode where applicable.

#7146 is merged here and should be landed first.

- Tests: 609, Assertions: 405, Skipped: 204.
+ Tests: 1498, Assertions: 787, Skipped: 711.

This vastly increases the tests that are used from HTML5lib-tests from 609 to 1498! It's helped to discover two issues:

Doctypes are not exposed via next_token so cannot be put into the tree

HTML, HEAD, and BODY tags should always be generated. This breaks many tests. Add the missing tags to the processed tree so tests pass

This reverts commit e1a4234.

This reverts commit 770acec.

github-actions · 2024-07-31T17:47:14Z

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

The Plugin and Theme Directories cannot be accessed within Playground.
All changes will be lost when closing a tab with a Playground instance.
All changes will be lost when refreshing the page.
A fresh instance is created each time the link below is clicked.
Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

tests/phpunit/tests/html-api/wpHtmlProcessorHtml5lib.php

Previously, the fragment parser in WP_HTML_Processor has only allowed creating a fragment with the `<body>` context. In this patch, any context node is allowed.

dmsnell · 2024-08-02T00:39:36Z

src/wp-includes/html-api/class-wp-html-tag-processor.php

+				strlen( $this->sought_tag_name ) !== $this->tag_name_length ||
+				0 !== substr_compare( $this->html, $this->sought_tag_name, $this->tag_name_starts_at, $this->tag_name_length, true )
+			)
+		) {


we need this. next_tag( 'PLAINTEXT' ) stopped on P

dmsnell · 2024-08-02T01:57:45Z

@sirreal I've added active format reconstruction to the PLAINTEXT element, even though it's missing from the spec, but all the major parsers clearly perform this. I've submitted a PR to the HTML spec to add the rule.

whatwg/html#10540

dmsnell · 2024-08-05T21:49:57Z

Leaving a note for posterity's sake: adding active format reconstruction to PLAINTEXT is wrong. Adding PLAINTEXT is more complicated than it originally seemed, because even though every remaining token is a character token, if there are any character tokens left, they may trigger active format reconstruction. This is something I'll try and fix

dmsnell · 2024-08-05T22:39:29Z

Fragment support extracted into #7141 where I thought I had already done so, sorry!

sirreal · 2024-08-06T07:21:34Z

Thanks, I'll revert the changes to support arbitrary fragments here and continue to only support body. That work is more complicated and can continue in #7141.

This reverts commit a385685.

This is no longer required. The full or fragment parser will be used as necessary.

The "</#" HTML test demonstrates a bug that must be fixed.

A state change was missing when the input is too short to find a comment closer in an opened funky comment. This fixes a issue where `</#` does not correctly report incomplete input, but `</# ` does.

…t-initial-mode

github-actions · 2024-08-06T11:18:23Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props jonsurrell, dmsnell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

tests/phpunit/tests/html-api/wpHtmlProcessorHtml5lib.php

dmsnell

This is a nice improvement in how many tests run, and it's not trivial to fix the test runner. Thanks again @sirreal!

This is a tests-only change.

Previously the `html5lib` tests have only run in the fragment parser mode, assuming IN BODY context. This limited the number of tests which could run and was a result of the HTML Processor only supporting the IN BODY fragment parser. In [58836], however, a full parser was added to the HTML Processor. In this patch the full parser is utilized in order to run more of the previously-skipped tests, asserting more behaviors in the HTML parsing. Developed in #7117 Discussed in https://core.trac.wordpress.org/ticket/61646 Props: dmsnell, jonsurrell. See #61646. git-svn-id: https://develop.svn.wordpress.org/trunk@58859 602fd350-edb4-49c9-b593-d223f7449a82

dmsnell · 2024-08-06T22:06:52Z

Merged in [58859]
9928cd6

Previously the `html5lib` tests have only run in the fragment parser mode, assuming IN BODY context. This limited the number of tests which could run and was a result of the HTML Processor only supporting the IN BODY fragment parser. In [58836], however, a full parser was added to the HTML Processor. In this patch the full parser is utilized in order to run more of the previously-skipped tests, asserting more behaviors in the HTML parsing. Developed in WordPress/wordpress-develop#7117 Discussed in https://core.trac.wordpress.org/ticket/61646 Props: dmsnell, jonsurrell. See #61646. Built from https://develop.svn.wordpress.org/trunk@58859 git-svn-id: http://core.svn.wordpress.org/trunk@58255 1a063a9b-81f0-0310-95a4-ce76da25c4cd

Previously the `html5lib` tests have only run in the fragment parser mode, assuming IN BODY context. This limited the number of tests which could run and was a result of the HTML Processor only supporting the IN BODY fragment parser. In [58836], however, a full parser was added to the HTML Processor. In this patch the full parser is utilized in order to run more of the previously-skipped tests, asserting more behaviors in the HTML parsing. Developed in WordPress/wordpress-develop#7117 Discussed in https://core.trac.wordpress.org/ticket/61646 Props: dmsnell, jonsurrell. See #61646. Built from https://develop.svn.wordpress.org/trunk@58859 git-svn-id: https://core.svn.wordpress.org/trunk@58255 1a063a9b-81f0-0310-95a4-ce76da25c4cd

Previously the `html5lib` tests have only run in the fragment parser mode, assuming IN BODY context. This limited the number of tests which could run and was a result of the HTML Processor only supporting the IN BODY fragment parser. In [58836], however, a full parser was added to the HTML Processor. In this patch the full parser is utilized in order to run more of the previously-skipped tests, asserting more behaviors in the HTML parsing. Developed in WordPress#7117 Discussed in https://core.trac.wordpress.org/ticket/61646 Props: dmsnell, jonsurrell. See #61646. git-svn-id: https://develop.svn.wordpress.org/trunk@58859 602fd350-edb4-49c9-b593-d223f7449a82

sirreal added 15 commits July 31, 2024 19:33

HTML5Lib: enable head tests

ed6d1c7

HTML5Lib: Use full parser when context not provided

e83b01a

HTML5Lib: Strip doctypes from expected output

301c935

Doctypes are not exposed via next_token so cannot be put into the tree

HTML5Lib: Ignore tests with known issues

fd8dd1f

HTML5Lib: Handle PI lookalike comments

53b044f

HTML5Lib: Handle funky comments in tree construction

6fe4d93

PICKME: Bugfix on ?-initial invalid comment texts

f4805f7

HTML5Lib: Add special handling for missing html, head, body tags

f09a026

HTML, HEAD, and BODY tags should always be generated. This breaks many tests. Add the missing tags to the processed tree so tests pass

Finish skipping tests

28ba1a7

Revert "PICKME: Bugfix on ?-initial invalid comment texts"

cd6e126

This reverts commit e1a4234.

Revert "fixup! PICKME: Bugfix on ?-initial invalid comment texts"

3315063

This reverts commit 770acec.

Disable some unimplemented tests

1662852

Read the script-on flag and ignore tests

3805b1c

Fix up ignores

f673e0a

Test ignores cleanup

1c834ba

sirreal force-pushed the html-api/support-initial-mode branch from 4a97cab to 1c834ba Compare July 31, 2024 17:33

Lints

b5df8df

sirreal mentioned this pull request Jul 31, 2024

Tests and bugfixes for initial mode support dmsnell/wordpress-develop#16

Closed

sirreal commented Aug 1, 2024

View reviewed changes

tests/phpunit/tests/html-api/wpHtmlProcessorHtml5lib.php Show resolved Hide resolved

PICKME: Fix infinite loop in skip_script_data

f5ca994

dmsnell mentioned this pull request Aug 1, 2024

HTML API: Plans for 6.7 WordPress/gutenberg#60396

Closed

19 tasks

HTML API: Allow any fragment context.

a385685

Previously, the fragment parser in WP_HTML_Processor has only allowed creating a fragment with the `<body>` context. In this patch, any context node is allowed.

dmsnell reviewed Aug 2, 2024

View reviewed changes

dmsnell force-pushed the html-api/support-initial-mode branch from adfe657 to a385685 Compare August 5, 2024 22:19

Merge branch 'trunk' into html-api/support-initial-mode

bb178ab

sirreal added 7 commits August 6, 2024 09:22

Revert "HTML API: Allow any fragment context."

08eabad

This reverts commit a385685.

Remove SKIP_HEAD_TESTS code

89ce774

This is no longer required. The full or fragment parser will be used as necessary.

Remove skip for bug

49365af

The "</#" HTML test demonstrates a bug that must be fixed.

Add test case for bug

1de7514

Set state to incomplete on short HTML funky comments

c08c379

A state change was missing when the input is too short to find a comment closer in an opened funky comment. This fixes a issue where `</#` does not correctly report incomplete input, but `</# ` does.

Merge branch 'html-api/fix-unclosed-comment-bug' into html-api/suppor…

6563eec

…t-initial-mode

Remove unnecessary test skips

d0ad5c5

sirreal requested a review from dmsnell August 6, 2024 11:18

sirreal marked this pull request as ready for review August 6, 2024 11:18

sirreal commented Aug 6, 2024

View reviewed changes

tests/phpunit/tests/html-api/wpHtmlProcessorHtml5lib.php Outdated Show resolved Hide resolved

sirreal and others added 2 commits August 6, 2024 13:19

Improve comment where html, head, body tags are appended.

0645a3d

Merge branch 'trunk' into html-api/support-initial-mode

00fe961

dmsnell approved these changes Aug 6, 2024

View reviewed changes

dmsnell closed this Aug 6, 2024

dmsnell deleted the html-api/support-initial-mode branch August 6, 2024 22:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTML API: Use full parser for html5lib tests #7117

HTML API: Use full parser for html5lib tests #7117

sirreal commented Jul 31, 2024 •

edited

Loading

github-actions bot commented Jul 31, 2024

dmsnell Aug 2, 2024

dmsnell commented Aug 2, 2024

dmsnell commented Aug 5, 2024

dmsnell commented Aug 5, 2024

sirreal commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

dmsnell left a comment

dmsnell commented Aug 6, 2024

HTML API: Use full parser for html5lib tests #7117

HTML API: Use full parser for html5lib tests #7117

Conversation

sirreal commented Jul 31, 2024 • edited Loading

github-actions bot commented Jul 31, 2024

Test using WordPress Playground

Some things to be aware of

dmsnell Aug 2, 2024

Choose a reason for hiding this comment

dmsnell commented Aug 2, 2024

dmsnell commented Aug 5, 2024

dmsnell commented Aug 5, 2024

sirreal commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

dmsnell left a comment

Choose a reason for hiding this comment

dmsnell commented Aug 6, 2024

sirreal commented Jul 31, 2024 •

edited

Loading