[Perf] Linux/arm64: 7 Regressions on 11/15/2022 10:02:59 PM

[performanceautofiler]

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Replace_Char - Duration of single invocation	1.87 μs	9.09 μs	4.87	0.12	False
Replace_Char - Duration of single invocation	755.94 ns	3.01 μs	3.98	0.06	False
Replace_String - Duration of single invocation	1.40 μs	6.45 μs	4.60	0.19	False
Replace_Char - Duration of single invocation	341.01 ns	646.62 ns	1.90	0.06	False
Replace_Char - Duration of single invocation	17.97 μs	100.18 μs	5.57	0.31	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -127.937768973504 (T) = (0 -9076.428413868736) / Math.Sqrt((1.9274515033569084 / (2)) + (15865.153813640083 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.857230143893031 = (1868.6428571395736 - 9076.428413868736) / 1868.6428571395736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -125.96566780768231 (T) = (0 -2995.851386390025) / Math.Sqrt((137.042717116517 / (2)) + (1250.0843539342538 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.006961792053188 = (747.6615804851324 - 2995.851386390025) / 747.6615804851324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -114.23844465757479 (T) = (0 -6476.488507298945) / Math.Sqrt((1183.4285986790633 / (2)) + (6812.004895361375 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -3.540013448090606 = (1426.5350931995063 - 6476.488507298945) / 1426.5350931995063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'l', newChar: '!')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -78.52526110953517 (T) = (0 -648.9350022546917) / Math.Sqrt((0.009406659617197681 / (2)) + (76.8274977177608 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.9026006032518725 = (341.0778915688084 - 648.9350022546917) / 341.0778915688084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -68.1178161251841 (T) = (0 -101010.08698302243) / Math.Sqrt((434626.6658096842 / (2)) + (6427420.275974766 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -4.770229469673614 = (17505.38475356959 - 101010.08698302243) / 17505.38475356959 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentQueue - Duration of single invocation	57.67 μs	62.08 μs	1.08	0.19	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach<String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.ConcurrentQueue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.0728286662609 (T) = (0 -61936.88488338162) / Math.Sqrt((560009.1076157368 / (2)) + (362681.8130281565 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.0838808210473095 = (57143.63025958385 - 61936.88488338162) / 57143.63025958385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	0589ce6c3022f4b9b7571268be5f97f45a003ad7
Compare	48e43bf4150a20e55562d004b30c8209f7a9ec33
Diff	Diff

Regressions in System.Collections.Concurrent.Count<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Stack - Duration of single invocation	2.26 μs	2.42 μs	1.07	0.12	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.Count<Int32>*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.Count<Int32>.Stack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 1:30:37 AM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -31.44566734994971 (T) = (0 -2421.9770575233315) / Math.Sqrt((47.86769098962399 / (2)) + (3.7793899800192152 / (5))) is less than -2.5705818356363865 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (5) - 2, .025) and -0.06896222453952625 = (2265.7274522180983 - 2421.9770575233315) / 2265.7274522180983 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kotlarmilos]

Same as dotnet/runtime#78820