Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Added better error handling for the Boa tester #1984

Closed
wants to merge 1 commit into from

Conversation

Razican
Copy link
Member

@Razican Razican commented Mar 28, 2022

Trying to fix the issue in #1982, I noticed that we didn't have a proper error handling for the boa tester.

This adds the anyhow dependency to the tester, which makes it much easier to handle errors and bubble them up with attached context. Thanks to this I was able to easily find out the issue, and I think it could be useful to have it. It gives errors such as this one:

Error: could not read the suite test
    caused by: error reading sub-suite ./test262/test/built-ins
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm/WrappedFunction
    caused by: error reading test ./test262/test/built-ins/ShadowRealm/WrappedFunction/throws-typeerror-on-revoked-proxy.js
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3

@Razican Razican added enhancement New feature or request test Issues and PRs related to the tests. labels Mar 28, 2022
@Razican Razican added this to the v0.15.0 milestone Mar 28, 2022
@github-actions
Copy link

Test262 conformance changes

VM implementation

Test result main count PR count difference
Total 88,428 88,428 0
Passed 43,991 43,991 0
Ignored 21,495 21,495 0
Failed 22,942 22,942 0
Panics 0 0 0
Conformance 49.75% 49.75% 0.00%

@codecov
Copy link

codecov bot commented Mar 28, 2022

Codecov Report

Merging #1984 (d83f02f) into main (5d2420e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1984   +/-   ##
=======================================
  Coverage   45.93%   45.93%           
=======================================
  Files         206      206           
  Lines       17138    17138           
=======================================
  Hits         7873     7873           
  Misses       9265     9265           
Impacted Files Coverage Δ
boa_tester/src/main.rs 0.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d2420e...d83f02f. Read the comment docs.

@github-actions
Copy link

Benchmark for d345ee6

Click to view benchmark
Test Base PR %
Arithmetic operations (Compiler) 512.4±1.88ns 507.8±5.41ns -0.90%
Arithmetic operations (Execution) 677.0±0.89ns 679.5±0.68ns +0.37%
Arithmetic operations (Parser) 5.3±0.02µs 5.3±0.01µs 0.00%
Array access (Compiler) 1271.8±2.70ns 1269.7±2.49ns -0.17%
Array access (Execution) 7.4±0.02µs 7.4±0.07µs 0.00%
Array access (Parser) 11.4±0.01µs 11.5±0.02µs +0.88%
Array creation (Compiler) 1845.5±11.18ns 1864.0±12.10ns +1.00%
Array creation (Execution) 2.5±0.00ms 2.5±0.00ms 0.00%
Array creation (Parser) 13.1±0.03µs 13.1±0.02µs 0.00%
Array pop (Compiler) 3.8±0.01µs 3.8±0.05µs 0.00%
Array pop (Execution) 1110.7±6.47µs 1104.1±2.92µs -0.59%
Array pop (Parser) 129.7±0.10µs 130.5±0.37µs +0.62%
Boolean Object Access (Compiler) 1123.7±5.50ns 983.3±1.82ns -12.49%
Boolean Object Access (Execution) 3.8±0.01µs 3.8±0.01µs 0.00%
Boolean Object Access (Parser) 14.0±0.02µs 14.0±0.03µs 0.00%
Clean js (Compiler) 3.4±0.01µs 3.4±0.01µs 0.00%
Clean js (Execution) 692.7±3.63µs 688.4±4.11µs -0.62%
Clean js (Parser) 28.2±0.05µs 28.0±0.03µs -0.71%
Create Realm 267.4±0.27ns 266.6±5.74ns -0.30%
Dynamic Object Property Access (Compiler) 1604.7±5.78ns 1634.8±4.51ns +1.88%
Dynamic Object Property Access (Execution) 5.1±0.03µs 5.0±0.03µs -1.96%
Dynamic Object Property Access (Parser) 10.2±0.13µs 10.3±0.02µs +0.98%
Fibonacci (Compiler) 2.3±0.01µs 2.3±0.01µs 0.00%
Fibonacci (Execution) 1280.9±2.59µs 1280.2±1.83µs -0.05%
Fibonacci (Parser) 15.8±0.03µs 15.8±0.03µs 0.00%
For loop (Compiler) 1957.4±3.46ns 1956.6±5.17ns -0.04%
For loop (Execution) 15.2±0.05µs 15.1±0.04µs -0.66%
For loop (Parser) 13.5±0.13µs 13.4±0.03µs -0.74%
Mini js (Compiler) 3.3±0.04µs 3.3±0.01µs 0.00%
Mini js (Execution) 647.8±2.96µs 646.0±3.02µs -0.28%
Mini js (Parser) 24.7±0.04µs 24.7±0.04µs 0.00%
Number Object Access (Compiler) 1036.7±2.43ns 914.9±1.79ns -11.75%
Number Object Access (Execution) 8.9±0.04µs 2.9±0.01µs -67.42%
Number Object Access (Parser) 10.9±0.02µs 9.6±0.03µs -11.93%
Object Creation (Compiler) 4.2±0.03µs 1401.4±10.69ns -66.63%
Object Creation (Execution) 4.8±0.02µs 4.7±0.01µs -2.08%
Object Creation (Parser) 8.8±0.02µs 8.9±0.02µs +1.14%
RegExp (Compiler) 1619.2±4.30ns 1631.6±4.50ns +0.77%
RegExp (Execution) 11.3±0.03µs 11.4±0.03µs +0.88%
RegExp (Parser) 9.9±0.01µs 9.9±0.02µs 0.00%
RegExp Creation (Compiler) 1403.6±5.98ns 1386.9±9.92ns -1.19%
RegExp Creation (Execution) 25.5±0.20µs 8.5±0.03µs -66.67%
RegExp Creation (Parser) 8.2±0.09µs 8.2±0.02µs 0.00%
RegExp Literal (Compiler) 1640.1±4.35ns 1634.8±3.90ns -0.32%
RegExp Literal (Execution) 11.4±0.03µs 11.5±0.07µs +0.88%
RegExp Literal (Parser) 7.9±0.02µs 8.0±0.01µs +1.27%
RegExp Literal Creation (Compiler) 1407.5±5.73ns 1416.4±4.61ns +0.63%
RegExp Literal Creation (Execution) 8.5±0.03µs 8.6±0.03µs +1.18%
RegExp Literal Creation (Parser) 6.2±0.02µs 6.3±0.01µs +1.61%
Static Object Property Access (Compiler) 1423.2±4.88ns 1407.9±5.72ns -1.08%
Static Object Property Access (Execution) 4.9±0.01µs 4.9±0.01µs 0.00%
Static Object Property Access (Parser) 9.6±0.02µs 9.6±0.02µs 0.00%
String Object Access (Compiler) 1483.6±5.48ns 1485.3±6.10ns +0.11%
String Object Access (Execution) 5.6±0.01µs 5.6±0.02µs 0.00%
String Object Access (Parser) 13.7±0.02µs 13.7±0.02µs 0.00%
String comparison (Compiler) 2.1±0.01µs 2.1±0.01µs 0.00%
String comparison (Execution) 13.0±0.11µs 4.3±0.02µs -66.92%
String comparison (Parser) 10.7±0.02µs 10.6±0.01µs -0.93%
String concatenation (Compiler) 1624.1±3.31ns 1624.1±6.28ns 0.00%
String concatenation (Execution) 4.1±0.01µs 4.1±0.02µs 0.00%
String concatenation (Parser) 7.3±0.01µs 7.3±0.01µs 0.00%
String copy (Compiler) 1298.8±2.53ns 1292.1±3.03ns -0.52%
String copy (Execution) 11.6±0.15µs 3.8±0.05µs -67.24%
String copy (Parser) 5.4±0.01µs 4.8±0.01µs -11.11%
Symbols (Compiler) 907.3±3.70ns 914.0±4.31ns +0.74%
Symbols (Execution) 3.9±0.01µs 3.9±0.01µs 0.00%
Symbols (Parser) 4.2±0.01µs 4.2±0.01µs 0.00%

@HalidOdat
Copy link
Member

bors r+

bors bot pushed a commit that referenced this pull request Mar 29, 2022
Trying to fix the issue in #1982, I noticed that we didn't have a proper error handling for the boa tester.

This adds the `anyhow` dependency to the tester, which makes it much easier to handle errors and bubble them up with attached context. Thanks to this I was able to easily find out the issue, and I think it could be useful to have it. It gives errors such as this one:

```
Error: could not read the suite test
    caused by: error reading sub-suite ./test262/test/built-ins
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm/WrappedFunction
    caused by: error reading test ./test262/test/built-ins/ShadowRealm/WrappedFunction/throws-typeerror-on-revoked-proxy.js
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3
```
@bors
Copy link

bors bot commented Mar 29, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Added better error handling for the Boa tester [Merged by Bors] - Added better error handling for the Boa tester Mar 29, 2022
@bors bors bot closed this Mar 29, 2022
@bors bors bot deleted the test262_update branch March 29, 2022 04:03
Razican added a commit that referenced this pull request Jun 8, 2022
Trying to fix the issue in #1982, I noticed that we didn't have a proper error handling for the boa tester.

This adds the `anyhow` dependency to the tester, which makes it much easier to handle errors and bubble them up with attached context. Thanks to this I was able to easily find out the issue, and I think it could be useful to have it. It gives errors such as this one:

```
Error: could not read the suite test
    caused by: error reading sub-suite ./test262/test/built-ins
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm
    caused by: error reading sub-suite ./test262/test/built-ins/ShadowRealm/WrappedFunction
    caused by: error reading test ./test262/test/built-ins/ShadowRealm/WrappedFunction/throws-typeerror-on-revoked-proxy.js
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3
    caused by: while scanning a block scalar, found a tab character where an indentation space is expected at line 4 column 3
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants