Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to .NET Core 2.2 and compiled regexes #14

Merged
merged 1 commit into from
Nov 13, 2019

Conversation

danmoseley
Copy link
Contributor

.NET Core 2.0 is out of support, so moving to latest 2.2
.NET Core 2.1 added support for compiled regexes (same as .NET Framework) so I'm reenabling that. Although .NET still trails the pack, this roughly doubles performance.

@danmoseley
Copy link
Contributor Author

@mariomka?

@danmoseley
Copy link
Contributor Author

2.2 can be 3.0 now 😄

@mariomka
Copy link
Owner

I'm sorry. I'll try to review it all soon.

@mariomka mariomka merged commit f724ab9 into mariomka:master Nov 13, 2019
@mariomka
Copy link
Owner

Thank you! I have merged it but I will update to 3.0.

@danmoseley danmoseley deleted the 22 branch November 13, 2019 18:49
@danmoseley
Copy link
Contributor Author

thanks @mariomka

@danmoseley
Copy link
Contributor Author

It would be great if you have time to update the numbers in the README using 3.0 -- although as I mentioned above, I do not think .NET looks great, it is less bad. We will look at what we can do in the next version.

@mariomka
Copy link
Owner

I have updated the benchmark using 3.0.
Also, I have configured the matcher with RegexOptions.ECMAScript (see #7), it is a trick but it's ok.

@mariomka
Copy link
Owner

Maybe I should add a benchmark with and without RegexOptions.ECMAScript. What do you think?

@danmoseley
Copy link
Contributor Author

@mariomka does it make a significant difference? (is it slower?)

I see it rarely used in .NET code, so I think "real world" is mostly without it.

@mariomka
Copy link
Owner

With RegexOptions.ECMAScript it's faster, like 2x:

With:

C# .Net Core | 672.25 | 557.74 | 88.69 | 1318.68
C# Mono | 1877.98 | 1540.01 | 175.38 | 3593.36


Without:

C# .Net Core | 1312.87 | 1080.66 | 94.72 | 2488.24
C# Mono | 3152.46 | 2637.13 | 182.02 | 5971.60

But if it isn't a real use case then I will remove it.

@mariomka
Copy link
Owner

Updated.

@danmoseley
Copy link
Contributor Author

With RegexOptions.ECMAScript it's faster, like 2x:

Interesting. @eerhardt, FYI. In a quick look I only see one place where ECMAScript setting seems to be checked during execution (rather than parsing). I wonder whether it's avoiding some expensive backtracking.

@danmoseley
Copy link
Contributor Author

cc @stephentoub for the interesting observation about ECMASCript setting perf above.

@stephentoub
Copy link

In a quick look I only see one place where ECMAScript setting seems to be checked during execution (rather than parsing)

The occurrences in the parser are more impactful, not because of parsing performance but because of choices made during parsing that affect execution. For example, when you write \d, with ECMAScript that ends up getting parsed as the equivalent of [0-9], whereas with the default, it's parsed as the equivalent of \p{Nd}, which allows for any Unicode decimal digit number. Similarly, whereas \w with ECMAScript is the equivalent of [A-Za-z0-9], by default it's the equivalent of [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Lm}\p{Mn}\p{Nd}\p{Pc}]. In both cases, the former is then cheaper to evaluate at execution time.

For .NET 5, the difference should be drastically reduced, with both getting much faster and the difference between them shrinking significantly. For example, when I run the benchmarks on my machine with .NET Core 3.1, I get:

909.9265 - 92
746.9995 - 5301
66.6641 - 5

and then when I set ECMAScript, I get:

483.9201 - 92
378.8119 - 5301
65.8511 - 5

However, with recent bits from master, I get this without ECMAScript:

298.3488 - 92
200.5408 - 5301
12.7788 - 5

and this with ECMAScript.

232.1069 - 92
141.4052 - 5301
12.0243 - 5

So, there's still a difference, because we still need to do slightly more work in the Unicode category case just in case the input isn't ASCII, but the gap is much smaller.

(As an aside, the methodology used for the benchmarking isn't ideal if the goal is to measure steady-state performance. Right now it's invoking the benchmarked code just once, with the timing numbers including all of the time required to compile/JIT/etc. the regex. The numbers get much better on subsequent invocations.)

(Also, we're still working on improving regex perf in .NET 5, so I expect we'll see these numbers get even better.)

@mariomka
Copy link
Owner

mariomka commented Jan 7, 2020

It seems the improvement in .NET is awesome! Please, ping me when version 5.0 is released for updating results.

@danmoseley
Copy link
Contributor Author

@mariomka thanks for helping us make it better! Would you consider updating your benchmark to allow creating/compiling the regex before measurement begins? This is more realistic: most apps will use the same regex many times so parsing/compilation is heavily amortized .

@mariomka
Copy link
Owner

mariomka commented Jan 8, 2020

I'm sorry. at this moment I don't consider update the benchmarks. It's not perfect but it's enough to get an idea.

@danmoseley
Copy link
Contributor Author

Sure! Sounds good.

@danmoseley
Copy link
Contributor Author

danmoseley commented May 2, 2020

@stephentoub here are the results on my machine tfor 5.0 preview 3, with and without ECMAScript.

PHP | 27.26 | 25.49 | 8.69 | 61.44
C PCRE2 | 29.11 | 26.58 | 7.54 | 63.23
Rust | 31.56 | 31.06 | 8.77 | 71.39
C++ Boost | 66.54 | 66.11 | 24.88 | 157.53
Javascript | 100.98 | 70.18 | 2.25 | 173.41
Perl | 160.16 | 97.88 | 37.22 | 295.26
Crystal | 180.68 | 160.38 | 20.05 | 361.11
Dart | 151.68 | 140.46 | 122.32 | 414.46
C# .Net Core 50 ECMAScript | 220.78 | 173.74 | 17.42 | 411.93
D ldc | 232.31 | 230.48 | 7.00 | 469.79
C# .Net Core 50 | 337.16 | 270.67 | 18.56 | 626.39
D dmd | 313.51 | 325.64 | 9.52 | 648.66
Ruby | 380.72 | 339.94 | 62.01 | 782.67
Kotlin | 254.09 | 316.64 | 441.90 | 1012.63
Java | 248.69 | 343.29 | 449.65 | 1041.63
Python 2 | 336.48 | 218.90 | 528.54 | 1083.91
Go | 435.86 | 419.52 | 671.12 | 1526.49
C++ STL | 633.10 | 518.65 | 483.14 | 1634.89
C# .Net Core 31 | 1592.04 | 1334.50 | 113.59 | 3040.13
C# Mono | 4110.00 | 3490.05 | 229.06 | 7829.11

My machine is considerably slower than yours, based on the 3.1 numbers, so 5.0 seems to have improved a fair bit since your Jan numbers. But still a significant difference with and without ECMAScript

@stephentoub
Copy link

Are each of these finding the same number of matches on all of the inputs?

@danmoseley
Copy link
Contributor Author

Yup, logged that:

- Run
C PCRE2 running...........C PCRE2 ran. (counts 92, 5301, 5)
Crystal running...........Crystal ran. (counts 92, 5301, 5)
C++ STL running...........C++ STL ran. (counts 92, 5301, 5)
C++ Boost running...........C++ Boost ran. (counts 92, 5301, 5)
C# Mono running...........C# Mono ran. (counts 92, 5301, 5)
C# .Net Core running...........C# .Net Core ran. (counts 92, 5301, 5)
C# .Net Core 50 running...........C# .Net Core 50 ran. (counts 92, 5301, 5)
C# .Net Core 50 ECMA running...........C# .Net Core 50 ECMA ran. (counts 92, 5301, 5)
D dmd running...........D dmd ran. (counts 92, 5301, 5)
D ldc running...........D ldc ran. (counts 92, 5301, 5)
Dart running.Dart ran. (counts )
Dart Native running.sh: 1: dart/bin/benchmark: not found
Dart Native ran. (counts )
Go running...........Go ran. (counts 92, 5301, 5)
Java running...........Java ran. (counts 92, 5301, 5)
Javascript running...........Javascript ran. (counts 92, 5301, 5)
Kotlin running...........Kotlin ran. (counts 92, 5301, 5)
Perl running...........Perl ran. (counts 92, 5301, 5)
PHP running...........PHP ran. (counts 92, 5301, 5)
Python 2 running...........Python 2 ran. (counts 92, 5301, 5)
Python 3 running.sh: 1: python3.6: not found
Python 3 ran. (counts )
Python PyPy2 running.pypy2: error while loading shared libraries: libffi.so.6: cannot open shared object file: No such file or directory
Python PyPy2 ran. (counts )
Python PyPy3 running.sh: 1: pypy3: Too many levels of symbolic links
Python PyPy3 ran. (counts )
Ruby running...........Ruby ran. (counts 92, 5301, 5)
Rust running...........Rust ran. (counts 92, 5301, 5)

- Results
**PHP** | 26.57 | 24.27 | 8.20 | 59.05
**C PCRE2** | 29.40 | 25.70 | 7.15 | 62.25
**Rust** | 30.90 | 29.14 | 8.29 | 68.33
**C++ Boost** | 62.97 | 62.49 | 23.13 | 148.58
**Javascript** | 95.32 | 67.16 | 2.19 | 164.68
**Perl** | 151.18 | 90.58 | 33.98 | 275.74
**Crystal** | 172.62 | 148.42 | 19.01 | 340.04
**C# .Net Core 50 ECMA** | 215.35 | 172.11 | 17.35 | 404.80
**D ldc** | 223.99 | 221.51 | 6.71 | 452.21
**C# .Net Core 50** | 314.79 | 254.98 | 17.40 | 587.17
**D dmd** | 292.91 | 303.55 | 8.84 | 605.29
**Ruby** | 365.49 | 324.14 | 59.95 | 749.58
**Kotlin** | 237.89 | 300.62 | 408.42 | 946.93
**Java** | 233.96 | 317.21 | 429.29 | 980.46
**Python 2** | 317.74 | 210.41 | 521.91 | 1050.06
**Go** | 411.11 | 399.17 | 634.96 | 1445.24
**C++ STL** | 604.55 | 488.13 | 422.20 | 1514.88
**C# .Net Core** | 1504.76 | 1256.17 | 104.65 | 2865.57
**C# Mono** | 3923.71 | 3301.79 | 218.56 | 7444.07

@danmoseley
Copy link
Contributor Author

danmoseley commented May 3, 2020

BTW, I tried with/without ECMAScript on regexredux, and it has a negligible effect, as expected as it's using trivial ASCII character classes, instead of \w.

@danmoseley
Copy link
Contributor Author

The #21 PR removed unicode support for Rust, to match PCRE. I also noticed Java has Unicode disabled. I believe we should be fair - I opened #26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants