Skip to content
This repository has been archived by the owner on Dec 18, 2018. It is now read-only.

Manifest feature request: for tests that could be manual or reftest, support a preference for reftests #196

Closed
qyearsley-zz opened this issue Mar 31, 2017 · 1 comment · Fixed by web-platform-tests/wpt#5341

Comments

@qyearsley-zz
Copy link

For example the test css/css-writing-modes-3/background-position-vrl-018.xht contains the following:

  <link rel="match" href="background-position-vrl-018-ref.xht" />

  <meta content="image interact" name="flags" />

Because of the "interact" flag, the manifest generator now considers this to be a manual test, even though it could also be a reference test.

Might we want to change the behavior of the manifest to treat this as a reftest, or provide an option to treat it as a reftest?

Changing it to always treat such tests as reftests when possible would be an easy, simple change; it would just involve changing the order of the elif blocks at SourceFile.manifest_items

More context: http://crbug.com/706953

@mrego
Copy link
Member

mrego commented Apr 4, 2017

We should find a way to fix this as it's making harder to import tests and run them automatically into Blink.

If the suggestion from @qyearsley is not good enough. Another option would be to fix the tests by changing the flags removing interact when it's not really mandatory.
From the doc (dunno if interact has some other meaning):

interact: Requires human interaction (such as for testing scrolling behavior)

For example, in some tests that require focus we've a small JavaScript that focus the element, but we still have them marked as interact: css/selectors4/focus-within-001.html
But human interaction is not required to run these tests. Only in the case that you don't have JavaScript support, then human interaction would be needed. But is this something we want to consider yet?

Pinging some people with more experience on CSS test suite to check what they think about this: @fantasai, @frivoal, @gsnedders. Thanks!

mrego added a commit to mrego/wpt that referenced this issue Apr 4, 2017
There were a bunch of tests that have both a reference file and
the "interact" flag.
They use some small JavaScript to execute some steps
before checking the output against reference file.
This caused them to be wrongly considered manual tests
when they can be run automatically.

For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
mrego added a commit to mrego/wpt that referenced this issue Apr 5, 2017
There were a bunch of tests that have both a reference file and
the "interact" flag.
They use some small JavaScript to execute some steps
before checking the output against reference file.
This caused them to be wrongly considered manual tests
when they can be run automatically.

For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
mrego added a commit to mrego/wpt that referenced this issue Apr 7, 2017
There were a bunch of tests that have both a reference file and
the "interact" flag.
They use some small JavaScript to execute some steps
before checking the output against reference file.
This caused them to be wrongly considered manual tests
when they can be run automatically.

For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants