Feature Request: Disregard or elsehow mark tentative tests.

[lukebjerring]

From @otherdaniel on August 25, 2017 9:14

Thanks for the dashboard!

Please add some feature to disregard tentative tests (or maybe account for them separately), so that implementing a proposed feature won't count "against" browsers.

Example: I am currently doing a first draft implementation of extending SRI (subresource integrity) with digital signatures, as proposed to (but not ratified by) the W3C webappsec group. To support that work, I have added webplatform tests for this feature, in the subresource-integrity dir. The dashboard now makes it look like browser are behind on implementing this, despite it merely being a proposed feature.

References:

• Explainer for feature in W3C webappsec:
  https://github.com/w3c/webappsec-subresource-integrity/blob/master/signature-based-restrictions-explainer.markdown
• Tentative WPT tests for this feature on the wptdashboard: http://wpt.fyi/subresource-integrity

Also, maybe related: Chrome/Chromium seems to run wpt tests with experimental features enabled, while your dashboard runs the release config. That's probably the better choice; but making both, release + experimental result sets available might be useful.

Copied from original issue: web-platform-tests/results-collection#99

[lukebjerring]

From @jgraham on August 25, 2017 10:37

FWIW I think this should be a non-goal. The point of the dashboard is to give knowledgable participants (i.e. browser vnedors) insights into where interop is lacking, and improvements they should make. It's not intended to be used for marketing comparisons, so it doesn't matter if implementations look "behind" on a spec that is in the early stages but still has testcases.

[lukebjerring]

From @foolip on October 9, 2017 13:19

I think we should treat tentative tests as different in the kinds of metrics we're discussing in web-platform-tests/results-collection#83 and in any per-browser reports (for prioritization purposes) that are produced.

Tentative tests are ones where the best shape of the spec is still unknown, and someone is trying to figure it out by doing more research, shipping things, etc. I think that at least to begin with, we want to apply roughly zero pressure on odd-one-out scenarios involving tentative tests. Rather, we need a regular per-directory triage of tentative tests to figure out which can be made non-tentative.

[lukebjerring]

From @foolip on October 9, 2017 13:24

Also, maybe related: Chrome/Chromium seems to run wpt tests with experimental features enabled, while your dashboard runs the release config. That's probably the better choice; but making both, release + experimental result sets available might be useful.

Yep, I agree. Looks like we don't have an existing issue for this, so: web-platform-tests/results-collection#146

[foolip]

With recent discussion on searching/filtering (@mdittmer) what comes to mind is an is:tentative query thing which we could use to exclude these tests from any view we like, and in summary numbers and graphs as well. It would be based simply on whether ".tentative." was part of the test name, so we could use that condition directly as well.

[stephenmcgruer]

Having implemented this in #1611, it occurs to me in testing it that this is really replicating existing behavior:

"is:tentative" == ".tentative."
"!is:tentative" == "none(.tentative.)"

Do we think it provides enough ergonomics value to include anyway? I'm leaning towards 'yes', but only very slightly. @Hexcles @foolip for their thoughts.

[foolip]

I think it is worth it, not least because it’s more discoverable and can be included in autocomplete suggestions.