Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Ignore wastefull RegExp tests #1840

Closed
wants to merge 1 commit into from

Conversation

raskad
Copy link
Member

@raskad raskad commented Feb 15, 2022

With the implementation of String.fromCodePoint in #1123 some RegExp tests are now running for a long time. These tests check every unicode codepoint for all regexp property escape/character classes.

This not only makes the developer experience significantly worse, but also wastes cpu resources for the benefit of "completeness". I think these tests are completely useless. Ironically the unicode tables in the tests are generated - from the same data, that the unicode tables in the regex engine are also generated.

262 suite runtime:
Before: ~03:30 https://github.com/boa-dev/boa/runs/5191567446?check_suite_focus=true
After: ~31:00 https://github.com/boa-dev/boa/runs/5196405437?check_suite_focus=true

@raskad raskad added test Issues and PRs related to the tests. Internal Category for changelog labels Feb 15, 2022
@raskad raskad added this to the v0.14.0 milestone Feb 15, 2022
@github-actions
Copy link

Test262 conformance changes

VM implementation

Test result main count PR count difference
Total 87,912 87,912 0
Passed 41,036 40,956 -80
Ignored 19,929 21,153 +1,224
Failed 26,947 25,803 -1,144
Panics 0 0 0
Conformance 46.68% 46.59% -0.09%

@codecov
Copy link

codecov bot commented Feb 15, 2022

Codecov Report

Merging #1840 (97c2b19) into main (be26b10) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1840   +/-   ##
=======================================
  Coverage   55.67%   55.67%           
=======================================
  Files         201      201           
  Lines       17375    17375           
=======================================
  Hits         9674     9674           
  Misses       7701     7701           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be26b10...97c2b19. Read the comment docs.

@github-actions
Copy link

Benchmark for 05eeb00

Click to view benchmark
Test Base PR %
Arithmetic operations (Compiler) 390.7±0.75ns 396.5±0.83ns +1.48%
Arithmetic operations (Execution) 1740.6±21.06ns 1741.6±39.45ns +0.06%
Arithmetic operations (Parser) 5.7±0.04µs 5.7±0.04µs 0.00%
Array access (Compiler) 841.3±3.92ns 818.2±1.82ns -2.75%
Array access (Execution) 11.0±0.09µs 10.9±0.07µs -0.91%
Array access (Parser) 12.8±0.09µs 12.9±0.10µs +0.78%
Array creation (Compiler) 1226.6±4.71ns 1214.6±9.65ns -0.98%
Array creation (Execution) 3.2±0.01ms 3.2±0.01ms 0.00%
Array creation (Parser) 14.3±0.10µs 14.2±0.11µs -0.70%
Array pop (Compiler) 2.7±0.01µs 2.7±0.01µs 0.00%
Array pop (Execution) 1397.1±4.47µs 1392.1±4.47µs -0.36%
Array pop (Parser) 140.4±0.15µs 140.7±0.27µs +0.21%
Boolean Object Access (Compiler) 692.2±1.43ns 690.4±2.46ns -0.26%
Boolean Object Access (Execution) 6.8±0.09µs 7.0±0.08µs +2.94%
Boolean Object Access (Parser) 15.0±0.02µs 15.0±0.02µs 0.00%
Clean js (Compiler) 2.2±0.01µs 2.3±0.01µs +4.55%
Clean js (Execution) 1490.4±13.05µs 1497.4±9.60µs +0.47%
Clean js (Parser) 30.5±0.11µs 30.7±0.12µs +0.66%
Create Realm 334.5±0.25ns 335.3±0.24ns +0.24%
Dynamic Object Property Access (Compiler) 1219.2±6.24ns 1189.3±9.35ns -2.45%
Dynamic Object Property Access (Execution) 6.9±0.04µs 6.8±0.10µs -1.45%
Dynamic Object Property Access (Parser) 11.5±0.04µs 11.7±0.06µs +1.74%
Fibonacci (Compiler) 1465.0±3.11ns 1436.6±2.88ns -1.94%
Fibonacci (Execution) 2.9±0.01ms 2.9±0.01ms 0.00%
Fibonacci (Parser) 17.2±0.05µs 17.4±0.05µs +1.16%
For loop (Compiler) 1325.7±3.98ns 1334.5±8.68ns +0.66%
For loop (Execution) 42.1±0.13µs 42.1±0.14µs 0.00%
For loop (Parser) 14.9±0.06µs 15.1±0.06µs +1.34%
Mini js (Compiler) 2.2±0.01µs 2.2±0.01µs 0.00%
Mini js (Execution) 1384.0±8.04µs 1388.2±9.54µs +0.30%
Mini js (Parser) 26.7±0.06µs 26.6±0.04µs -0.37%
Number Object Access (Compiler) 638.3±8.37ns 633.1±1.69ns -0.81%
Number Object Access (Execution) 5.4±0.07µs 5.3±0.03µs -1.85%
Number Object Access (Parser) 11.6±0.02µs 11.6±0.02µs 0.00%
Object Creation (Compiler) 983.3±2.95ns 980.3±3.62ns -0.31%
Object Creation (Execution) 6.3±0.03µs 6.1±0.04µs -3.17%
Object Creation (Parser) 10.1±0.06µs 10.1±0.04µs 0.00%
RegExp (Compiler) 1252.9±80.74ns 1212.8±7.84ns -3.20%
RegExp (Execution) 12.7±0.03µs 12.7±0.03µs 0.00%
RegExp (Parser) 11.0±0.04µs 11.0±0.06µs 0.00%
RegExp Creation (Compiler) 1093.3±2.61ns 1092.1±6.90ns -0.11%
RegExp Creation (Execution) 9.6±0.06µs 9.7±0.04µs +1.04%
RegExp Creation (Parser) 9.1±0.04µs 9.1±0.05µs 0.00%
RegExp Literal (Compiler) 1202.1±44.41ns 1204.3±12.84ns +0.18%
RegExp Literal (Execution) 12.7±0.03µs 12.7±0.03µs 0.00%
RegExp Literal (Parser) 8.9±0.04µs 8.9±0.06µs 0.00%
RegExp Literal Creation (Compiler) 1081.8±4.82ns 1045.2±3.00ns -3.38%
RegExp Literal Creation (Execution) 9.7±0.05µs 9.6±0.05µs -1.03%
RegExp Literal Creation (Parser) 7.0±0.04µs 7.0±0.04µs 0.00%
Static Object Property Access (Compiler) 1007.9±3.57ns 999.3±8.81ns -0.85%
Static Object Property Access (Execution) 6.5±0.04µs 6.3±0.07µs -3.08%
Static Object Property Access (Parser) 10.8±0.05µs 10.8±0.03µs 0.00%
String Object Access (Compiler) 1048.8±6.11ns 1041.9±8.97ns -0.66%
String Object Access (Execution) 8.9±0.06µs 9.0±0.09µs +1.12%
String Object Access (Parser) 14.6±0.03µs 14.9±0.22µs +2.05%
String comparison (Compiler) 1368.5±4.46ns 1328.0±4.38ns -2.96%
String comparison (Execution) 7.4±0.03µs 7.4±0.03µs 0.00%
String comparison (Parser) 11.7±0.02µs 12.0±0.25µs +2.56%
String concatenation (Compiler) 1092.2±4.19ns 1072.0±7.74ns -1.85%
String concatenation (Execution) 6.3±0.03µs 6.3±0.04µs 0.00%
String concatenation (Parser) 8.1±0.05µs 8.2±0.17µs +1.23%
String copy (Compiler) 909.3±3.47ns 900.3±2.82ns -0.99%
String copy (Execution) 5.3±0.03µs 5.3±0.03µs 0.00%
String copy (Parser) 6.1±0.06µs 6.2±0.06µs +1.64%
Symbols (Compiler) 643.8±1.60ns 640.2±1.52ns -0.56%
Symbols (Execution) 5.0±0.03µs 4.9±0.03µs -2.00%
Symbols (Parser) 4.7±0.05µs 4.7±0.06µs 0.00%

Copy link
Member

@RageKnify RageKnify left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just ran test262 on main and on this code and the difference was huge:
main = 21m13s
ignore-wastefull-regexp-tests = 59s
Definitely seems like a good choice.

Copy link
Member

@Razican Razican left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with this, but at some point we should run those tests, maybe before releases, or when we improve their performance.

@Razican
Copy link
Member

Razican commented Feb 16, 2022

bors r+

bors bot pushed a commit that referenced this pull request Feb 16, 2022
With the implementation of `String.fromCodePoint` in #1123 some `RegExp` tests are now running for a long time. These tests check every unicode codepoint for all regexp property escape/character classes.

This not only makes the developer experience significantly worse, but also wastes cpu resources for the benefit of "completeness". I think these tests are completely useless. Ironically the unicode tables in the tests are generated - from the same data, that the unicode tables in the regex engine are also generated.

262 suite runtime:
Before: ~03:30 https://github.com/boa-dev/boa/runs/5191567446?check_suite_focus=true
After: ~31:00 https://github.com/boa-dev/boa/runs/5196405437?check_suite_focus=true
@bors
Copy link

bors bot commented Feb 16, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Ignore wastefull RegExp tests [Merged by Bors] - Ignore wastefull RegExp tests Feb 16, 2022
@bors bors bot closed this Feb 16, 2022
@bors bors bot deleted the ignore-wastefull-regexp-tests branch February 16, 2022 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Internal Category for changelog test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants