test262 conformance as an line item the compat table

[littledan]

Test262 is another test suite for JavaScript. It provides a number of very detailed set of around 20k tests and aims to test every line in the specification, going into some details which the current compat table omits. However, there is not currently a place where test262 scores are surfaced, and test262 does not include categorization into features, weighting, or a pretty website.

For me as an implementer, test262 has been extremely useful in identifying non-compliant parts of V8 so that we can fix them. Although many of the bugs don't reflect something that seems likely for a user to run into, it feels to me like a good tool for making a JavaScript implementation into a reliable, consistent surface.

Maybe it would be useful for others as well to see in a comparison context. I'm wondering whether it might make sense to have one item in the compat table here be for the entire test262 score. It could be factored into the overall score, or left to the side for independent reference. Thoughts?

[ljharb]

I love this idea. This table is mostly checking presence and mostly not checking compliance, and compliance matters far more money than presence.

[kangax]

This is a great idea! The fact that test262 compliance is not listed anywhere on a web for any browser seems like an unfortunate omission. We've pushed the table far enough, to the point where some engines score 100%. It's only fair to expose even more detailed metrics. I imagine a separate table would be most convenient for implementors? Something with breakdown by test group, to see what exactly is failing?

[ljharb]

If there is a way to automatically convert commandline test262 results to a web-consumable format, that would be ideal imo, because then we could programmatically show a breakdown with collapsible details etc.

[natorion]

There is a web test runner for test262. It produces html and json as a result. You can have a look at it at http://v8.github.io/test262/website/default.html. The runner unfortunately is quite buggy (did not work in Safari the last time I checked).

[littledan]

Yes, there's significant technical work needed to make this work, which @jugglinmike, @natorion and I have looked into to some extent. If anyone is interested in implementation along these lines, I could help you get started; I won't have the time to do this technical work myself at the moment.

[bterlson]

I support this idea, although I still hope someone will contribute a better website to ECMA like the old test262.ecmascript.org website. I think the hardest problem with this is going to be the UI - there is a lot of data in test262 and presenting it in a useful way is hard.

FWIW, I have just released a new version of https://github.com/bterlson/test262-harness which supports running test262 in browsers from the command line.

[chicoxyzzy]

This also might be useful
https://github.com/bakkot/test262-web-runner/

[chicoxyzzy]

Just some related useful links:
https://github.com/bterlson/eshost-cli
babel/babylon#654