Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New test262 results format #783

Merged
merged 2 commits into from
Oct 3, 2020
Merged

New test262 results format #783

merged 2 commits into from
Oct 3, 2020

Conversation

Razican
Copy link
Member

@Razican Razican commented Oct 3, 2020

This PR improves the format of the Test262 suite results, in order to be able to fully implement #769. This also reduces the payload size for the website, and removes the "info" file.

@Razican Razican added enhancement New feature or request test Issues and PRs related to the tests. labels Oct 3, 2020
@Razican Razican added this to the v0.11.0 milestone Oct 3, 2020
@codecov
Copy link

codecov bot commented Oct 3, 2020

Codecov Report

Merging #783 into master will decrease coverage by 0.03%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #783      +/-   ##
==========================================
- Coverage   59.28%   59.25%   -0.04%     
==========================================
  Files         156      155       -1     
  Lines        9820     9825       +5     
==========================================
  Hits         5822     5822              
- Misses       3998     4003       +5     
Impacted Files Coverage Δ
boa_tester/src/exec.rs 0.00% <0.00%> (ø)
boa_tester/src/main.rs 0.00% <ø> (ø)
boa_tester/src/results.rs 0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e60d2c...0beb265. Read the comment docs.

@github-actions
Copy link

github-actions bot commented Oct 3, 2020

Benchmark for 996fe12

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 328.3±16.74ns 347.4±13.45ns -5.50%
Arithmetic operations (Full) 245.4±8.38µs 244.5±10.14µs +0.37%
Array access (Execution) 8.1±0.03µs 7.4±0.32µs +9.46%
Array access (Full) 257.6±9.79µs 255.9±12.12µs +0.66%
Array creation (Execution) 3.3±0.02ms 3.0±0.12ms +10.00%
Array creation (Full) 3.2±0.16ms 3.4±0.15ms -5.88%
Array pop (Execution) 1173.0±5.57µs 1081.8±58.63µs +8.43%
Array pop (Full) 1425.4±84.69µs 1583.3±73.79µs -9.97%
Boolean Object Access (Execution) 4.3±0.20µs 4.6±0.14µs -6.52%
Boolean Object Access (Full) 252.0±12.19µs 249.5±10.50µs +1.00%
Clean js (Execution) 642.2±30.71µs 677.1±26.22µs -5.15%
Clean js (Full) 986.7±29.41µs 958.1±41.15µs +2.99%
Clean js (Parser) 32.3±1.39µs 31.9±1.87µs +1.25%
Create Realm 457.1±4.71ns 428.2±18.92ns +6.75%
Dynamic Object Property Access (Execution) 5.4±0.06µs 5.0±0.26µs +8.00%
Dynamic Object Property Access (Full) 247.1±12.37µs 255.4±13.77µs -3.25%
Expression (Parser) 6.5±0.25µs 6.3±0.38µs +3.17%
Fibonacci (Execution) 789.9±1.67µs 725.8±26.38µs +8.83%
Fibonacci (Full) 972.9±48.17µs 960.0±48.14µs +1.34%
For loop (Execution) 22.0±0.08µs 19.8±0.93µs +11.11%
For loop (Full) 271.8±12.61µs 260.2±12.54µs +4.46%
For loop (Parser) 15.8±0.69µs 16.4±0.55µs -3.66%
Goal Symbols (Parser) 10.6±0.43µs 10.7±0.58µs -0.93%
Hello World (Parser) 2.8±0.14µs 2.8±0.11µs 0.00%
Long file (Parser) 676.0±26.75ns 686.6±30.26ns -1.54%
Mini js (Execution) 595.3±33.20µs 602.4±31.89µs -1.18%
Mini js (Full) 899.6±35.90µs 875.9±41.23µs +2.71%
Mini js (Parser) 28.7±1.21µs 28.7±1.40µs 0.00%
Number Object Access (Execution) 3.6±0.17µs 3.5±0.16µs +2.86%
Number Object Access (Full) 249.2±12.39µs 251.4±12.01µs -0.88%
Object Creation (Execution) 4.5±0.03µs 4.1±0.18µs +9.76%
Object Creation (Full) 239.0±12.02µs 249.1±11.13µs -4.05%
RegExp (Execution) 63.3±1.23µs 59.0±3.36µs +7.29%
RegExp (Full) 328.7±15.00µs 322.9±15.03µs +1.80%
RegExp Literal (Execution) 64.4±1.90µs 63.7±2.60µs +1.10%
RegExp Literal (Full) 308.1±14.46µs 317.5±17.45µs -2.96%
RegExp Literal Creation (Execution) 63.6±1.27µs 61.1±2.88µs +4.09%
RegExp Literal Creation (Full) 314.1±12.45µs 312.1±15.65µs +0.64%
Static Object Property Access (Execution) 4.7±0.03µs 4.3±0.19µs +9.30%
Static Object Property Access (Full) 242.3±10.75µs 255.7±10.08µs -5.24%
String Object Access (Execution) 6.2±0.33µs 6.8±0.17µs -8.82%
String Object Access (Full) 251.0±10.58µs 250.2±12.81µs +0.32%
String comparison (Execution) 5.8±0.30µs 5.9±0.34µs -1.69%
String comparison (Full) 244.2±15.27µs 246.6±13.49µs -0.97%
String concatenation (Execution) 4.7±0.25µs 5.0±0.24µs -6.00%
String concatenation (Full) 245.1±13.81µs 238.1±12.19µs +2.94%
String copy (Execution) 3.5±0.18µs 3.5±0.18µs 0.00%
String copy (Full) 240.9±11.45µs 244.3±10.95µs -1.39%
Symbols (Execution) 3.1±0.01µs 2.9±0.13µs +6.90%
Symbols (Full) 242.1±13.26µs 235.6±9.35µs +2.76%

@github-actions
Copy link

github-actions bot commented Oct 3, 2020

Benchmark for 8a1ace0

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 328.2±15.98ns 349.7±15.68ns -6.15%
Arithmetic operations (Full) 241.2±6.57µs 233.5±11.01µs +3.30%
Array access (Execution) 7.5±0.39µs 7.4±0.31µs +1.35%
Array access (Full) 248.9±15.41µs 257.7±12.09µs -3.41%
Array creation (Execution) 3.0±0.13ms 3.1±0.13ms -3.23%
Array creation (Full) 3.2±0.13ms 3.3±0.14ms -3.03%
Array pop (Execution) 1047.8±45.05µs 1085.4±60.12µs -3.46%
Array pop (Full) 1491.1±70.60µs 1534.1±59.26µs -2.80%
Boolean Object Access (Execution) 4.4±0.17µs 4.4±0.18µs 0.00%
Boolean Object Access (Full) 258.0±8.35µs 243.8±11.85µs +5.82%
Clean js (Execution) 643.4±34.38µs 681.4±18.92µs -5.58%
Clean js (Full) 959.5±30.86µs 948.5±33.80µs +1.16%
Clean js (Parser) 30.9±1.64µs 31.9±1.34µs -3.13%
Create Realm 430.1±21.10ns 419.6±21.21ns +2.50%
Dynamic Object Property Access (Execution) 4.7±0.22µs 4.9±0.19µs -4.08%
Dynamic Object Property Access (Full) 262.7±10.77µs 247.8±12.47µs +6.01%
Expression (Parser) 6.3±0.22µs 6.0±0.28µs +5.00%
Fibonacci (Execution) 699.2±38.33µs 740.9±26.31µs -5.63%
Fibonacci (Full) 982.2±41.75µs 1003.6±35.86µs -2.13%
For loop (Execution) 19.9±1.03µs 20.7±0.81µs -3.86%
For loop (Full) 263.0±14.17µs 278.0±15.48µs -5.40%
For loop (Parser) 15.5±0.69µs 15.7±0.69µs -1.27%
Goal Symbols (Parser) 10.7±0.43µs 10.6±0.56µs +0.94%
Hello World (Parser) 2.8±0.09µs 2.6±0.13µs +7.69%
Long file (Parser) 679.2±36.26ns 675.3±31.20ns +0.58%
Mini js (Execution) 582.3±33.59µs 622.9±21.08µs -6.52%
Mini js (Full) 885.2±34.42µs 865.6±34.33µs +2.26%
Mini js (Parser) 27.4±1.39µs 28.2±1.52µs -2.84%
Number Object Access (Execution) 3.5±0.18µs 3.5±0.14µs 0.00%
Number Object Access (Full) 252.1±8.23µs 243.0±11.71µs +3.74%
Object Creation (Execution) 3.9±0.19µs 4.1±0.19µs -4.88%
Object Creation (Full) 245.3±10.28µs 255.9±10.26µs -4.14%
RegExp (Execution) 57.0±2.96µs 57.5±2.54µs -0.87%
RegExp (Full) 328.3±21.22µs 315.5±15.91µs +4.06%
RegExp Literal (Execution) 59.7±3.00µs 62.1±3.63µs -3.86%
RegExp Literal (Full) 314.9±16.58µs 315.2±14.60µs -0.10%
RegExp Literal Creation (Execution) 55.5±2.93µs 58.4±3.02µs -4.97%
RegExp Literal Creation (Full) 304.0±16.33µs 310.6±13.15µs -2.12%
Static Object Property Access (Execution) 4.3±0.22µs 4.4±0.18µs -2.27%
Static Object Property Access (Full) 252.5±9.87µs 240.9±11.55µs +4.82%
String Object Access (Execution) 6.3±0.31µs 6.4±0.22µs -1.56%
String Object Access (Full) 258.3±9.54µs 249.5±12.04µs +3.53%
String comparison (Execution) 5.8±0.29µs 5.6±0.32µs +3.57%
String comparison (Full) 260.7±8.17µs 249.6±12.91µs +4.45%
String concatenation (Execution) 4.6±0.23µs 4.8±0.30µs -4.17%
String concatenation (Full) 259.2±9.20µs 246.1±11.85µs +5.32%
String copy (Execution) 3.4±0.18µs 3.6±0.17µs -5.56%
String copy (Full) 253.2±9.42µs 236.4±9.80µs +7.11%
Symbols (Execution) 2.8±0.16µs 3.0±0.12µs -6.67%
Symbols (Full) 227.6±9.62µs 242.4±9.62µs -6.11%

@HalidOdat HalidOdat merged commit 2a509de into master Oct 3, 2020
@HalidOdat HalidOdat deleted the test262_improvements branch October 3, 2020 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants