[Perf] Linux/arm64: 7 Regressions on 11/15/2022 10:02:59 PM

[performanceautofiler]

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Replace_Char - Duration of single invocation	341.01 ns	646.62 ns	1.90	0.06	False
Replace_Char - Duration of single invocation	17.97 μs	100.18 μs	5.57	0.31	False
Replace_Char - Duration of single invocation	1.87 μs	9.09 μs	4.87	0.12	False
Replace_Char - Duration of single invocation	755.94 ns	3.01 μs	3.98	0.06	False
Replace_String - Duration of single invocation	1.40 μs	6.45 μs	4.60	0.19	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'l', newChar: '!')

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -78.52526110953517 (T) = (0 -648.9350022546917) / Math.Sqrt((0.009406659617197681 / (2)) + (76.8274977177608 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.9026006032518725 = (341.0778915688084 - 648.9350022546917) / 341.0778915688084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -68.1178161251841 (T) = (0 -101010.08698302243) / Math.Sqrt((434626.6658096842 / (2)) + (6427420.275974766 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -4.770229469673614 = (17505.38475356959 - 101010.08698302243) / 17505.38475356959 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -127.937768973504 (T) = (0 -9076.428413868736) / Math.Sqrt((1.9274515033569084 / (2)) + (15865.153813640083 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.857230143893031 = (1868.6428571395736 - 9076.428413868736) / 1868.6428571395736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -125.96566780768231 (T) = (0 -2995.851386390025) / Math.Sqrt((137.042717116517 / (2)) + (1250.0843539342538 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.006961792053188 = (747.6615804851324 - 2995.851386390025) / 747.6615804851324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -114.23844465757479 (T) = (0 -6476.488507298945) / Math.Sqrt((1183.4285986790633 / (2)) + (6812.004895361375 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.540013448090606 = (1426.5350931995063 - 6476.488507298945) / 1426.5350931995063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentQueue - Duration of single invocation	57.67 μs	62.08 μs	1.08	0.19	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach<String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.ConcurrentQueue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.0728286662609 (T) = (0 -61936.88488338162) / Math.Sqrt((560009.1076157368 / (2)) + (362681.8130281565 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.0838808210473095 = (57143.63025958385 - 61936.88488338162) / 57143.63025958385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Collections.Concurrent.Count<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Stack - Duration of single invocation	2.26 μs	2.42 μs	1.07	0.12	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.Count<Int32>*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.Count<Int32>.Stack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -31.44566734994971 (T) = (0 -2421.9770575233315) / Math.Sqrt((47.86769098962399 / (2)) + (3.7793899800192152 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.06896222453952625 = (2265.7274522180983 - 2421.9770575233315) / 2265.7274522180983 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kotlarmilos]

Data are missing, hard to identify, potentially in 3cfd6cf

[kotlarmilos]

@naricc do you think that it is related to your changes?

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[kotlarmilos]

Could be related to Regex changes in 4793d91...1a476d7

[stephentoub]

Could be related to Regex changes in 4793d91...1a476d7

What points to that? I don't see regex being used in any of these tests, and it's not used in the implementation of string or the collections.

[kotlarmilos]

You are right, the tests don't rely on regex. It is hard to tell due to missing data. After reviewing changes in the BCL it might be related to MemoryExtension changes as mentioned in dotnet/perf-autofiling-issues#9819.

[kotlarmilos]

Probably mitigated by a6962a7