Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 23 Regressions on 10/18/2022 8:25:03 PM #9173

Closed
performanceautofiler bot opened this issue Oct 25, 2022 · 5 comments
Closed

[Perf] Linux/x64: 23 Regressions on 10/18/2022 8:25:03 PM #9173

performanceautofiler bot opened this issue Oct 25, 2022 · 5 comments
Assignees
@kotlarmilos
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Oct 25, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 295.62 ns 460.27 ns 1.56 0.04 False
Span - Duration of single invocation 385.50 ns 572.40 ns 1.48 0.03 False
ReadOnlySpan - Duration of single invocation 460.52 ns 686.27 ns 1.49 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.IterateFor<String>.Array(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 460.27212915945285 > 310.40094856820406.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -1645.1981764227107 (T) = (0 -460.20773144191634) / Math.Sqrt((0.08473447622811564 / (19)) + (0.13326729340804813 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.5569386523087086 = (295.58501278100886 - 460.20773144191634) / 295.58501278100886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateFor&lt;String&gt;.Span(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 572.4040367868025 > 405.1100695000064.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 9/24/2022 1:05:07 PM, 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -1152.4979835041881 (T) = (0 -572.271538325635) / Math.Sqrt((0.32729382455991274 / (19)) + (0.21492135248395466 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.48337219929478525 = (385.7909286675997 - 572.271538325635) / 385.7909286675997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateFor&lt;String&gt;.ReadOnlySpan(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 686.2685567369641 > 483.5681325538998.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -1568.0777352423747 (T) = (0 -684.7104306737283) / Math.Sqrt((0.18309522957066454 / (20)) + (0.2702917777498113 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.48639646520766255 = (460.65127757019275 - 684.7104306737283) / 460.65127757019275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayGet - Duration of single invocation 75.67 ns 96.76 ns 1.28 0.03 False
BitArrayGet - Duration of single invocation 8.03 μs 10.71 μs 1.33 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 4)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 96.75768493473021 > 79.55610772768277.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 9/2/2022 11:09:40 AM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 9/24/2022 1:05:07 PM, 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -303.47767199379564 (T) = (0 -97.24591856073397) / Math.Sqrt((0.06051503597573693 / (20)) + (0.04834166461479053 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.2846026319976783 = (75.70116714575572 - 97.24591856073397) / 75.70116714575572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 10.70907591894977 > 8.439377721559463.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 9/2/2022 11:09:40 AM, 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -2625.954277808161 (T) = (0 -10708.85761619122) / Math.Sqrt((11.359415963395355 / (19)) + (10.468297173543583 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.3321828058632666 = (8038.579667188981 - 10708.85761619122) / 8038.579667188981 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 47.91 ns 51.40 ns 1.07 0.04 False
ToString - Duration of single invocation 3.46 ns 4.49 ns 1.30 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 51.40447517002075 > 50.4560042156166.
IsChangePoint: Marked as a change because one of 8/31/2022 10:21:55 AM, 9/26/2022 7:45:50 PM, 10/11/2022 2:16:14 AM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -20.385626284288545 (T) = (0 -52.39254420862988) / Math.Sqrt((0.37566934757479803 / (20)) + (0.46791121773896427 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.0823999570516665 = (48.40405237204662 - 52.39254420862988) / 48.40405237204662 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ToString(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.486759191167195 > 3.6350650135907507.
IsChangePoint: Marked as a change because one of 9/24/2022 1:05:07 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -15.211051180514566 (T) = (0 -4.550370560334728) / Math.Sqrt((0.07933745222977286 / (19)) + (0.014210890861341044 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.3000830567475713 = (3.5000614281663114 - 4.550370560334728) / 3.5000614281663114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsDefinedClassHitInherit - Duration of single invocation 430.65 ns 522.33 ns 1.21 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Reflection.Attributes.IsDefinedClassHitInherit

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 522.3346842732565 > 448.72629184082524.
IsChangePoint: Marked as a change because one of 8/31/2022 4:31:58 AM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -22.507408002773804 (T) = (0 -497.2816975566382) / Math.Sqrt((68.97497720029791 / (20)) + (139.2215850445575 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.1596259701338553 = (428.8293901345079 - 497.2816975566382) / 428.8293901345079 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in Benchstone.BenchI.Fib

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 174.77 μs 190.23 μs 1.09 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.Fib*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

Benchstone.BenchI.Fib.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 190.22624870912222 > 183.49310976335474.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/31/2022 10:21:55 AM, 9/2/2022 11:09:40 AM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -84.39156479440398 (T) = (0 -190292.52500139896) / Math.Sqrt((538383.9756171878 / (20)) + (137578.32828228996 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.0871176790750394 = (175043.1702695765 - 190292.52500139896) / 175043.1702695765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
PopCount_ulong - Duration of single invocation 745.58 ns 1.10 μs 1.48 0.02 False
PopCount_uint - Duration of single invocation 667.95 ns 1.10 μs 1.64 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_BitOperations.PopCount_ulong

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.1029670920770624 > 782.5058521054318.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -2018.9008154973928 (T) = (0 -1103.334609569032) / Math.Sqrt((0.24564818432092128 / (20)) + (0.46015656907611835 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.48045372210438164 = (745.2678817955241 - 1103.334609569032) / 745.2678817955241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.PopCount_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.097771333859895 > 701.1181891581643.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -1789.9288248782145 (T) = (0 -1098.7432604155429) / Math.Sqrt((0.19794355566088884 / (20)) + (1.1543505359999602 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.6455902355755014 = (667.6894628213971 - 1098.7432604155429) / 667.6894628213971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 4.89 μs 5.33 μs 1.09 0.02 False
Span - Duration of single invocation 389.84 ns 572.44 ns 1.47 0.03 False
IEnumerable - Duration of single invocation 3.05 μs 3.27 μs 1.07 0.02 False
Dictionary - Duration of single invocation 3.57 μs 4.09 μs 1.14 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.IterateForEach<String>.Stack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.329221890949303 > 5.128999434320685.
IsChangePoint: Marked as a change because one of 8/1/2022 12:35:11 AM, 9/8/2022 6:17:28 PM, 9/10/2022 11:12:52 PM, 10/3/2022 8:45:37 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -39.44242311764368 (T) = (0 -5328.486637849531) / Math.Sqrt((1545.244741564552 / (19)) + (837.5320970725099 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.08672204990710436 = (4903.265410235324 - 5328.486637849531) / 4903.265410235324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.Span(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 572.4405990948634 > 409.6104602191342.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 9/24/2022 1:05:07 PM, 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/5/2022 3:28:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -2250.5755171036476 (T) = (0 -571.9171124532262) / Math.Sqrt((0.04122527359276266 / (19)) + (0.10479964996831563 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.46659506181946253 = (389.96252431376934 - 571.9171124532262) / 389.96252431376934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.IEnumerable(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 3.27051434441553 > 3.204006509257588.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/31/2022 4:31:58 AM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -173.67969907343294 (T) = (0 -3272.572480552448) / Math.Sqrt((16.06624676837605 / (20)) + (19.081375028155634 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.07192196846651651 = (3052.9950657081554 - 3272.572480552448) / 3052.9950657081554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.088642901835976 > 3.610246054018882.
IsChangePoint: Marked as a change because one of 8/17/2022 9:10:20 PM, 9/26/2022 2:03:14 AM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -18.229420077440402 (T) = (0 -3885.783626370847) / Math.Sqrt((1340.901368781897 / (20)) + (11858.9588583496 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.125026987001266 = (3453.9470352869625 - 3885.783626370847) / 3453.9470352869625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 3.61 ns 4.72 ns 1.31 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Byte*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Byte.ToString(value: 0)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.724658541414403 > 3.7974213763211067.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -21.996721926478187 (T) = (0 -4.675785260214927) / Math.Sqrt((0.028795092038519002 / (19)) + (0.023255108105412182 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.30631687258513685 = (3.5793652813821337 - 4.675785260214927) / 3.5793652813821337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MeasureReverseUsingNtoH - Duration of single invocation 1.43 μs 2.19 μs 1.53 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.MeasureReverseUsingNtoH

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.1907415392110696 > 1.5037210973548802.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 9/2/2022 11:09:40 AM, 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -844.0894903561467 (T) = (0 -2192.0902639383744) / Math.Sqrt((10.657795147464059 / (19)) + (5.951317385472404 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.5298044163665323 = (1432.921908504389 - 2192.0902639383744) / 1432.921908504389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Oct 25, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in Benchstone.BenchI.Ackermann

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 5.28 μs 5.72 μs 1.08 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.Ackermann*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

Benchstone.BenchI.Ackermann.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.716145565249 > 5.566366944152928.
IsChangePoint: Marked as a change because one of 8/1/2022 12:35:11 AM, 8/31/2022 4:31:58 AM, 10/5/2022 3:28:28 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -33.61498004102284 (T) = (0 -5730.4965972134405) / Math.Sqrt((1619.1151565273606 / (20)) + (1777.345761523444 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.07878781462526285 = (5311.977498748478 - 5730.4965972134405) / 5311.977498748478 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObjInt - Duration of single invocation 703.13 μs 775.39 μs 1.10 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

PerfLabTests.CastingPerf2.CastingPerf.ObjInt

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 775.389178125 > 740.8733144423172.
IsChangePoint: Marked as a change because one of 9/2/2022 11:09:40 AM, 9/29/2022 8:01:59 PM, 10/5/2022 3:28:28 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -46.95178280400355 (T) = (0 -774578.2501037217) / Math.Sqrt((19447481.290160667 / (20)) + (28237696.91274744 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.09752447725588613 = (705750.3191549595 - 774578.2501037217) / 705750.3191549595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterfaceInterfaceMethod - Duration of single invocation 2.74 ms 2.98 ms 1.09 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethod

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.9768868029513884 > 2.8776795456770827.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 10/3/2022 8:45:37 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -192.88912415606333 (T) = (0 -2979686.491321591) / Math.Sqrt((27196866.23351698 / (20)) + (3927857.5070587476 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.0868410373529451 = (2741602.855352945 - 2979686.491321591) / 2741602.855352945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexerCheckLengthHoisting - Duration of single invocation 60.29 ns 90.58 ns 1.50 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_String.IndexerCheckLengthHoisting

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 90.58280590748495 > 63.27031448251626.
IsChangePoint: Marked as a change because one of 8/2/2022 6:54:34 PM, 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -561.4057491637192 (T) = (0 -90.10945193550621) / Math.Sqrt((0.027869088089585718 / (20)) + (0.03383999574264553 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.49226540060237123 = (60.38433371110288 - 90.10945193550621) / 60.38433371110288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 1.45 μs 1.68 μs 1.16 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.CreateAddAndClear<String>.Span(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.6796152364417987 > 1.5195644445613088.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 9/29/2022 8:01:59 PM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -33.14571098452713 (T) = (0 -1663.0031216628129) / Math.Sqrt((683.3928888970228 / (19)) + (284.23557356445616 / (24))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (24) - 2, .025) and -0.15984498968351793 = (1433.814980842043 - 1663.0031216628129) / 1433.814980842043 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e8fa4a736d6bdcfbbe2c2dc1dd1303214bb97085
Compare 51e3afb2ba42609c3711cb50fb5eaa6800bc1fde
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ctor_seeded - Duration of single invocation 527.39 ns 630.84 ns 1.20 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Random*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Random.ctor_seeded

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 630.840041147687 > 553.4370195574022.
IsChangePoint: Marked as a change because one of 9/6/2022 5:23:23 PM, 9/29/2022 8:01:59 PM, 10/11/2022 2:16:14 AM, 10/18/2022 9:26:37 AM, 10/25/2022 2:56:59 AM falls between 10/16/2022 1:15:28 PM and 10/25/2022 2:56:59 AM.
IsRegressionStdDev: Marked as regression because -167.1107628848186 (T) = (0 -631.0869171596652) / Math.Sqrt((5.242186658801211 / (20)) + (3.110744057023592 / (24))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (24) - 2, .025) and -0.19865491900848778 = (526.4959140047515 - 631.0869171596652) / 526.4959140047515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@naricc
Copy link

naricc commented Oct 26, 2022

This looks like the only mono change: dotnet/runtime@8a24ff0. I'm not sure how that could cause these regressions. @kotlarmilos any ideas?

@naricc naricc removed the untriaged label Oct 26, 2022
@naricc naricc mentioned this issue Oct 26, 2022
Closed
This was referenced Nov 2, 2022
Closed
Open
@radekdoulik radekdoulik mentioned this issue Nov 9, 2022
Open
@kotlarmilos
Copy link
Member

This looks like the only mono change: dotnet/runtime@8a24ff0. I'm not sure how that could cause these regressions. @kotlarmilos any ideas?

No regressions by dotnet/runtime@8a24ff0 in local environment (OSX-arm64). I can check the others changes

@kotlarmilos
Copy link
Member

Opcodes for System.Collections.IterateFor<String>.Array(Size: 512) in dotnet/runtime@10fc8ae (parent commit)

Total executed opcodes 855639
68% Opcounts 590000, calls 10000, Method Program:Array (), imethod ptr 0x1439686f0
81% Opcounts 110788, calls 6, Method System.SpanHelpers:IndexOfNullCharacter (char&), imethod ptr 0x1438e56e0

Opcodes forSystem.Collections.IterateFor<String>.Array(Size: 512) in dotnet/runtime@8a24ff0

Total executed opcodes 855579
68% Opcounts 590000, calls 10000, Method Program:Array (), imethod ptr 0x10e028ef0
81% Opcounts 110788, calls 6, Method System.SpanHelpers:IndexOfNullCharacter (char&), imethod ptr 0x13f0886e0

@naricc
Copy link

naricc commented Nov 30, 2022

I don't think it is worth spending more time investigating this, so I am closing it out.

@naricc naricc closed this as completed Nov 30, 2022
@naricc naricc added the LinkedFromMaster Is this issue linked from the master perf issue label Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kotlarmilos kotlarmilos

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kotlarmilos @naricc