This repository has been archived by the owner on Dec 18, 2018. It is now read-only.
Prefer automated tests for css tests with interact or font flags #197
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Fixes #196 |
jgraham
force-pushed
the
css_manifest_order
branch
from
ec23c64 to
7884be0
Compare
|
I actually think we should change the tests instead to remove the "interact" flag since they're not interactive at all. |
For tests with "interact" flag that are also marked as reftests or testharness.js tests, prefer running the test automatically vs assuming it's a manual test. It has been verified that this is reasonable based on an insepction of the existing tests.
jgraham
force-pushed
the
css_manifest_order
branch
from
7884be0 to
d673bdc
Compare
Codecov Report
@@ Coverage Diff @@
## master #197 +/- ##
==========================================
+ Coverage 87.75% 89.38% +1.62%
==========================================
Files 24 24
Lines 2418 2722 +304
Branches 406 505 +99
==========================================
+ Hits 2122 2433 +311
+ Misses 230 223 -7
Partials 66 66
Continue to review full report at Codecov.
|
|
@annevk Sure, happy to accept a patch to do that. I still think this is a reasonable change to take. |
|
Once web-platform-tests/wpt#5341 lands, this will misrun 829 tests that require fonts to be installed. |
|
I would strongly, strongly prefer just fixing the metadata. |
mrego
added a commit
to mrego/wpt
that referenced
this pull request
Apr 4, 2017
There were a bunch of tests that have both a reference file and the "interact" flag. They use some small JavaScript to execute some steps before checking the output against reference file. This caused them to be wrongly considered manual tests when they can be run automatically. For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
mrego
added a commit
to mrego/wpt
that referenced
this pull request
Apr 5, 2017
There were a bunch of tests that have both a reference file and the "interact" flag. They use some small JavaScript to execute some steps before checking the output against reference file. This caused them to be wrongly considered manual tests when they can be run automatically. For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
mrego
added a commit
to mrego/wpt
that referenced
this pull request
Apr 7, 2017
There were a bunch of tests that have both a reference file and the "interact" flag. They use some small JavaScript to execute some steps before checking the output against reference file. This caused them to be wrongly considered manual tests when they can be run automatically. For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For tests with "interact" or "font" flags that are also marked as
reftests or testharness.js tests, prefer running the test
automatically vs assuming it's a manual test. For "interact" flags it
has been verified that this is reasonable. For the "font" flag this is
likely to produce some false positives (things detected as tests that
are not actually able to run correctly without installing some other
font), but the status quo is to not run many tests that use
@font-face, so this is an improvement, even though it's not quite right.
This change is