Arm64: Regressions in System.Text.RegularExpressions. #64381

performanceautofiler · 2022-01-27T07:00:52Z

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Count - Duration of single invocation	196.24 μs	300.98 μs	1.53	0.08	True
Count - Duration of single invocation	63.20 μs	154.12 μs	2.44	0.01	True
Count - Duration of single invocation	100.61 μs	197.45 μs	1.96	0.01	True
Count - Duration of single invocation	100.27 μs	193.24 μs	1.93	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -93.30490056184595 (T) = (0 -302110.5269860348) / Math.Sqrt((13223906.599727279 / (22)) + (2489855.7165398365 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5188910589732372 = (198902.03790537818 - 302110.5269860348) / 198902.03790537818 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -46.070117601034106 (T) = (0 -154706.8570747855) / Math.Sqrt((10781.494298185917 / (22)) + (15759608.222075848 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -1.4457364834676718 = (63255.73426268532 - 154706.8570747855) / 63255.73426268532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -204.59814678741992 (T) = (0 -197075.21528389637) / Math.Sqrt((339001.03073299903 / (22)) + (817712.6447385071 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9484295773909126 = (101145.6701184932 - 197075.21528389637) / 101145.6701184932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -147.49859779935338 (T) = (0 -194205.28421047283) / Math.Sqrt((1011372.6929403255 / (22)) + (1410671.2860712947 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9213449191127261 = (101077.78269200957 - 194205.28421047283) / 101077.78269200957 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.24 ms	5.02 ms	1.55	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -142.63817697387185 (T) = (0 -5004262.43697479) / Math.Sqrt((1469273523.1041913 / (22)) + (334968873.4781656 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5377666955395907 = (3254240.35485359 - 5004262.43697479) / 3254240.35485359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

ghost · 2022-01-27T16:07:37Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Count - Duration of single invocation	196.24 μs	300.98 μs	1.53	0.08	True
Count - Duration of single invocation	63.20 μs	154.12 μs	2.44	0.01	True
Count - Duration of single invocation	100.61 μs	197.45 μs	1.96	0.01	True
Count - Duration of single invocation	100.27 μs	193.24 μs	1.93	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -93.30490056184595 (T) = (0 -302110.5269860348) / Math.Sqrt((13223906.599727279 / (22)) + (2489855.7165398365 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5188910589732372 = (198902.03790537818 - 302110.5269860348) / 198902.03790537818 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -46.070117601034106 (T) = (0 -154706.8570747855) / Math.Sqrt((10781.494298185917 / (22)) + (15759608.222075848 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -1.4457364834676718 = (63255.73426268532 - 154706.8570747855) / 63255.73426268532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -204.59814678741992 (T) = (0 -197075.21528389637) / Math.Sqrt((339001.03073299903 / (22)) + (817712.6447385071 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9484295773909126 = (101145.6701184932 - 197075.21528389637) / 101145.6701184932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -147.49859779935338 (T) = (0 -194205.28421047283) / Math.Sqrt((1011372.6929403255 / (22)) + (1410671.2860712947 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9213449191127261 = (101077.78269200957 - 194205.28421047283) / 101077.78269200957 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.24 ms	5.02 ms	1.55	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -142.63817697387185 (T) = (0 -5004262.43697479) / Math.Sqrt((1469273523.1041913 / (22)) + (334968873.4781656 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5377666955395907 = (3254240.35485359 - 5004262.43697479) / 3254240.35485359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`, `untriaged`
Milestone:	-

ghost · 2022-01-27T16:07:58Z

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Count - Duration of single invocation	196.24 μs	300.98 μs	1.53	0.08	True
Count - Duration of single invocation	63.20 μs	154.12 μs	2.44	0.01	True
Count - Duration of single invocation	100.61 μs	197.45 μs	1.96	0.01	True
Count - Duration of single invocation	100.27 μs	193.24 μs	1.93	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -93.30490056184595 (T) = (0 -302110.5269860348) / Math.Sqrt((13223906.599727279 / (22)) + (2489855.7165398365 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5188910589732372 = (198902.03790537818 - 302110.5269860348) / 198902.03790537818 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -46.070117601034106 (T) = (0 -154706.8570747855) / Math.Sqrt((10781.494298185917 / (22)) + (15759608.222075848 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -1.4457364834676718 = (63255.73426268532 - 154706.8570747855) / 63255.73426268532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -204.59814678741992 (T) = (0 -197075.21528389637) / Math.Sqrt((339001.03073299903 / (22)) + (817712.6447385071 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9484295773909126 = (101145.6701184932 - 197075.21528389637) / 101145.6701184932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -147.49859779935338 (T) = (0 -194205.28421047283) / Math.Sqrt((1011372.6929403255 / (22)) + (1410671.2860712947 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.9213449191127261 = (101077.78269200957 - 194205.28421047283) / 101077.78269200957 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	4d16d0934259b2b1b056823986bb0ad08578c5c0
Compare	82ff4f89e5997ade30ac1a9de8f343c321dd10a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.24 ms	5.02 ms	1.55	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/25/2022 5:52:36 PM, 1/27/2022 2:00:00 AM falls between 1/11/2022 6:41:22 PM and 1/27/2022 2:00:00 AM.
IsRegressionStdDev: Marked as regression because -142.63817697387185 (T) = (0 -5004262.43697479) / Math.Sqrt((1469273523.1041913 / (22)) + (334968873.4781656 / (4))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (4) - 2, .025) and -0.5377666955395907 = (3254240.35485359 - 5004262.43697479) / 3254240.35485359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	`arch-arm64`, `area-System.Text.RegularExpressions`, `os-windows`, `tenet-performance`, `tenet-performance-benchmarks`, `area-CodeGen-coreclr`, `untriaged`
Milestone:	-

kunalspathak · 2022-01-27T16:08:07Z

Introduced by #63285

stephentoub · 2022-01-27T16:21:11Z

cc: @EgorBo

EgorBo · 2022-01-27T16:32:42Z

Looking at it now, might be Vector128.ExtractMostSignBits problem on ARM

stephentoub · 2022-01-27T16:59:03Z

@kunalspathak, is this only on ARM? I was actually expecting the cited benchmarks to get better, not worse. Do we see any differences on other platforms?

EgorBo · 2022-01-27T17:01:23Z

@kunalspathak, is this only on ARM? I was actually expecting the cited benchmarks to get better, not worse. Do we see any differences on other platforms?

We only triaged ARM issues today, usually it takes more days for auto-perf-filer to recognize regressions/improvements. We should wait till Tuesday's Perf-Triage (x64)

EgorBo · 2022-02-01T16:06:51Z

x64 regressions
dotnet/perf-autofiling-issues#3202
dotnet/perf-autofiling-issues#3212
dotnet/perf-autofiling-issues#3234
dotnet/perf-autofiling-issues#3241
dotnet/perf-autofiling-issues#3252

x64 Improvements:
dotnet/perf-autofiling-issues#3217
dotnet/perf-autofiling-issues#3237
dotnet/perf-autofiling-issues#3247
dotnet/perf-autofiling-issues#3246
dotnet/perf-autofiling-issues#3259
dotnet/perf-autofiling-issues#3206

arm64 regressions:
dotnet/perf-autofiling-issues#3284

arm64 improvements:
dotnet/perf-autofiling-issues#3286
dotnet/perf-autofiling-issues#3291

EgorBo · 2022-02-06T13:21:18Z

I took a look at couple of regressions and it feels like it's the worst possible case for the new algorithm - huge input (594915 chars) and we search for a value that is never found there
plain IndexOf works much better in this case - it doesn't need two vectors (e.g. new algorithm does two memory loads, two compare instructions, it does And instruction, its loop body is way bigger. Also, the new algorithm doesn't align data first (it matters for really large inputs)

I'll check whether data alignment will help or will change the algorithm to always use IndexOf first and start only if we found something

UPD data alignment won't work, one of the vectors will almost always be unaligned

stephentoub · 2022-02-06T14:50:21Z

huge input (594915 chars) and we search for a value that is never found there
plain IndexOf works much better in this case

Hmm. Isn't that part of the goal of this algorithm, to do well in this case? e.g. where there's frequently a 'T' in the input but rarely followed by 'n' four chars later?

EgorBo · 2022-02-06T15:30:46Z

@stephentoub
So we have 0.5Mb of text and we look for "zqj" - there are only 151 z chars in that text it means that with the new algorithm we waste a lot of time loading the 2nd vector from memory and doing the second compare. We also don't align data the way it's done in single-char IndexOf (it's not suitable for this algorithm).

The old algorithm does stop 151 times and does SequenceEquals but it's not a lot for 0.5mb data.

Unfortunately, we also can't just use the single-char IndexOf first and then fallback to the new algorithm in case of false positive - because as I said - there are 151 z chars in that text, we'll early fallback on the first z we meet (and in that case, the first z comes up in the first 100 chars)

EgorBo · 2022-02-06T15:39:50Z

So basically the new algorithm is almost always faster except very large data where the old one doesn't stop often.

The best we can do is always use the old one for very large data, at least we won't regress anybody (but also won't benefit for cases where the old one will have to stop very often in that large data)

danmoseley · 2022-02-06T16:59:31Z

In large data where the old algorithm wouldn't stop often, are there even then circumstances where the new algorithm would win - where the search pattern was long and ultimately did not match, perhaps?

stephentoub · 2022-02-06T17:25:04Z

very large data

Why is the "very large" relevant here? isn't it really about the density of matches of the first element, regardless of length (above the vectorized threshold)?

Also, some of the regressions involved characters that are presumably much more frequent, like 'S'?

EgorBo · 2022-02-06T19:51:57Z

In large data where the old algorithm wouldn't stop often, are there even then circumstances where the new algorithm would win - where the search pattern was long and ultimately did not match, perhaps?

Yes, if the value is also large it gives the new algorithm an advantage.

isn't it really about the density of matches of the first element

You're right 👍

Also, some of the regressions involved characters that are presumably much more frequent, like 'S'?

Well, there are 837 'S' symbols in that text (594915 char symbols) leading to 'S' per 711 chars on average = 88 LoadVectors for the new algorithm

EgorBo · 2022-02-06T20:13:18Z

Btw, 'S' case is optimized with #64872
at the same time 's' (lowercased) if far more popular - 27128 chars. Now let's see what will happen with sherlock:

Method	Job	Toolchain	Pattern	Options	Mean	Error	StdDev	Ratio	RatioSD
Count	Job-FSENCF	\Core_Root\corerun.exe	Sherlock	None	47.18 us	0.924 us	1.466 us	1.00	0.00
Count	Job-AJCOLR	\Core_Root_PR\corerun.exe	Sherlock	None	46.37 us	0.317 us	0.281 us	0.98	0.03

Count	Job-FSENCF	\Core_Root\corerun.exe	sherlock	None	578.40 us	3.485 us	2.911 us	1.00	0.00
Count	Job-AJCOLR	\Core_Root_PR\corerun.exe	sherlock	None	35.43 us	0.300 us	0.266 us	0.06	0.00

Sherlock is more or less the same perf, sherlock the new algorithm is 17 times faster!!

\Core_Root_PR\corerun.exe - is #64872
\Core_Root\corerun.exe - Main, but with SpanHelpers reverted to the state before the new algorithm.

EgorBo · 2022-06-11T12:08:36Z

Regressions mostly fixed (with help of #65632 and #64872 and probably some Regex-related PRs) e.g.

Some benchmarks are still slightly slower due to falling into "worst case" of the algorithm discussed above

kunalspathak transferred this issue from dotnet/perf-autofiling-issues Jan 27, 2022

dotnet-issue-labeler bot added area-System.Text.RegularExpressions untriaged New issue has not been triaged by the area owner labels Jan 27, 2022

kunalspathak changed the title ~~[Perf] Changes at 1/25/2022 9:10:05 PM~~ Arm64: Regressions in System.Text.RegularExpressions. Jan 27, 2022

kunalspathak added arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jan 27, 2022

kunalspathak assigned EgorBo Jan 27, 2022

kunalspathak mentioned this issue Jan 27, 2022

[Perf] Changes at 1/25/2022 9:10:05 PM dotnet/perf-autofiling-issues#3126

Closed

JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Jan 27, 2022

JulieLeeMSFT added this to the 7.0.0 milestone Jan 27, 2022

danmoseley removed the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jan 29, 2022

EgorBo mentioned this issue Feb 6, 2022

Improvements for SpanHelpers.IndexOf #64872

Merged

EgorBo mentioned this issue Feb 11, 2022

System.Text.RegularExpressions performance regressions #65187

Closed

EgorBo closed this as completed Jun 11, 2022

ghost locked as resolved and limited conversation to collaborators Jul 11, 2022

Arm64: Regressions in System.Text.RegularExpressions. #64381

Arm64: Regressions in System.Text.RegularExpressions. #64381

Comments

performanceautofiler bot commented Jan 27, 2022

Run Information

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

Docs

ghost commented Jan 27, 2022

Run Information

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

Docs

ghost commented Jan 27, 2022

Run Information

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Repro

Payloads

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)

Description of detection logic

Docs

kunalspathak commented Jan 27, 2022

stephentoub commented Jan 27, 2022

EgorBo commented Jan 27, 2022

stephentoub commented Jan 27, 2022

EgorBo commented Jan 27, 2022

EgorBo commented Feb 1, 2022 • edited Loading

EgorBo commented Feb 6, 2022 • edited Loading

stephentoub commented Feb 6, 2022 • edited Loading

EgorBo commented Feb 6, 2022 • edited Loading

EgorBo commented Feb 6, 2022

danmoseley commented Feb 6, 2022

stephentoub commented Feb 6, 2022

EgorBo commented Feb 6, 2022

EgorBo commented Feb 6, 2022 • edited Loading

EgorBo commented Jun 11, 2022

EgorBo commented Feb 1, 2022 •

edited

Loading

EgorBo commented Feb 6, 2022 •

edited

Loading

stephentoub commented Feb 6, 2022 •

edited

Loading

EgorBo commented Feb 6, 2022 •

edited

Loading

EgorBo commented Feb 6, 2022 •

edited

Loading