Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieve feature-based results for Test262 runs #1645

Closed
Tracked by #820
Razican opened this issue Oct 7, 2021 · 2 comments
Closed
Tracked by #820

Retrieve feature-based results for Test262 runs #1645

Razican opened this issue Oct 7, 2021 · 2 comments
Assignees
Labels
documentation update documentation enhancement New feature or request test Issues and PRs related to the tests.
Milestone

Comments

@Razican
Copy link
Member

Razican commented Oct 7, 2021

The current implementation of the Tes262 compliance results page shows the compliance test suite by test suite, which is straightforward, since we run the tests suite by suite, test by test, and then we just collect the information about it.

It would be very nice to have the option to also see it feature by feature. Some tests, such as this one have a list of features (only one in this case). This information is already being stored in the MetaData information structure here.

In order to show these results, we first need to gather them. Currently, when we run a test suite, we get a SuiteResult, as you can see here. These results are generated from the TestResult each test returns, as it can be seen here. The definition of both structures can be seen in the main.rs file of the boa_tester project, here.

We should change the SuiteResult structure to include information on the features each test had. This would allow having feature coverage per test-suite. Since the top-most suite will include feature information for all the suites below it, we can store that information on a JSON file, as we do with the rest of the SuiteResult.

Note that we want to store this information in a separate JSON file, so we will need to ignore part of the SuiteResult structure when serializing it, and extract that information to a new structure that will be stored in a new JSON file. Similarly to what we do with the ReducedResultInfo structure. Note that there should be also results for "no features", for base language tests.

After this, we should be able to use that file in the results page.

@Razican Razican added enhancement New feature or request documentation update documentation test Issues and PRs related to the tests. labels Oct 7, 2021
@NorbertGarfield
Copy link
Contributor

@Razican, I've got two questions:

  1. Has anybody already taken this issue?
  2. I could not figure out which output format is expected. Currently I stick to the following:
{
    "c" : "",
    "u" : "0bccacda693ada2cd1736d35eb912b27291ac6ff",
    "n" : "types",
    "f" : "Proxy, Symbol, cross-realm"
}

Where 'n' stands for 'name' and 'f' means 'features'. Do I need to add the information about passed tests or features here?

@Razican
Copy link
Member Author

Razican commented Mar 27, 2022

@Razican, I've got two questions:

  1. Has anybody already taken this issue?

Nobody has taken this issue :) let me assign it to you

  1. I could not figure out which output format is expected. Currently I stick to the following:
{
    "c" : "",
    "u" : "0bccacda693ada2cd1736d35eb912b27291ac6ff",
    "n" : "types",
    "f" : "Proxy, Symbol, cross-realm"
}

Where 'n' stands for 'name' and 'f' means 'features'. Do I need to add the information about passed tests or features here?

This looks pretty good :) for simplicity, I would use an array for the features: ["Proxy", "Symbol", "cross-realm"]. The rest looks good, I think it doesn't need any extra information, and you can use the current structure :)

@Razican Razican added this to the v0.15.0 milestone Mar 27, 2022
@bors bors bot closed this as completed in 5a2703b Mar 28, 2022
Razican pushed a commit that referenced this issue Jun 8, 2022
This Pull Request fixes/closes #1645.

It changes the following:

- Add `features` field to `SuiteResult` structure
- Fetch features from `TestSuite` and propagate them via `SuiteResult`
- Add `FeaturesInfo` structure and serialize it to `features.json`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation update documentation enhancement New feature or request test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants