Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 101 Regressions on 12/8/2022 2:00:38 AM #10689

Closed
performanceautofiler bot opened this issue Dec 13, 2022 · 5 comments
Closed

[Perf] Linux/arm64: 101 Regressions on 12/8/2022 2:00:38 AM #10689

performanceautofiler bot opened this issue Dec 13, 2022 · 5 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 13, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 9.15 μs 10.43 μs 1.14 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -23.046201629213062 (T) = (0 -10550.408244171696) / Math.Sqrt((3735.5206208113823 / (5)) + (11889.581167652124 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.15369647431302313 = (9144.873438617391 - 10550.408244171696) / 9144.873438617391 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 77.90 μs 87.83 μs 1.13 0.03 False
SortedList - Duration of single invocation 1.16 ms 2.70 ms 2.33 0.05 False
List - Duration of single invocation 200.87 μs 645.19 μs 3.21 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<Int32>.Queue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.061005787324758 (T) = (0 -89785.1724644837) / Math.Sqrt((2090113.0279509048 / (5)) + (4123224.281684529 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.13826914117309638 = (78878.6845015858 - 89785.1724644837) / 78878.6845015858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.22146105791375 (T) = (0 -2740373.2319444446) / Math.Sqrt((195096625.65049347 / (5)) + (6612484126.850057 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.3459906010868639 = (1168109.211807954 - 2740373.2319444446) / 1168109.211807954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -37.86351421168533 (T) = (0 -630304.5485111988) / Math.Sqrt((472354.36405035725 / (5)) + (515187197.28820944 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.144620785959499 = (200438.9690882482 - 630304.5485111988) / 200438.9690882482 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CopyTo<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 2.08 μs 2.92 μs 1.40 0.08 False
Array - Duration of single invocation 2.28 μs 4.01 μs 1.76 0.08 False
ImmutableArray - Duration of single invocation 2.31 μs 4.04 μs 1.75 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<String>.List(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -16.946339569051496 (T) = (0 -2842.4463023370645) / Math.Sqrt((5862.585712717553 / (5)) + (2673.3012229843 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.34371566503264356 = (2115.362927072828 - 2842.4463023370645) / 2115.362927072828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.Array(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.79898773952997 (T) = (0 -3992.8548441087723) / Math.Sqrt((5588.0147929482855 / (5)) + (481.6012055125668 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.7277348263537293 = (2311.034530996536 - 3992.8548441087723) / 2311.034530996536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ImmutableArray(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.75524533593149 (T) = (0 -4013.7574360041335) / Math.Sqrt((5092.367469591867 / (5)) + (3065.9816588590597 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.7195855384387284 = (2334.142353655965 - 4013.7574360041335) / 2334.142353655965 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.IO.Tests.Perf_RandomAccess

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadScatterAsync - Duration of single invocation 29.76 ms 35.83 ms 1.20 0.31 False
ReadScatter - Duration of single invocation 591.04 μs 658.08 μs 1.11 0.03 False
ReadScatterAsync - Duration of single invocation 30.00 ms 36.02 ms 1.20 0.38 False
ReadScatter - Duration of single invocation 22.30 ms 26.00 ms 1.17 0.22 False
ReadScatter - Duration of single invocation 565.76 μs 662.90 μs 1.17 0.11 False
ReadScatter - Duration of single invocation 20.25 ms 25.55 ms 1.26 0.43 False
ReadScatterAsync - Duration of single invocation 922.85 μs 1.02 ms 1.10 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_RandomAccess*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 104857600, buffersSize: 65536, options: Asynchronous)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -5.7634799354437805 (T) = (0 -35370466.32944139) / Math.Sqrt((2165042861321.6682 / (5)) + (424951515044.11664 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.13592072097137 = (31138146.94672946 - 35370466.32944139) / 31138146.94672946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 1048576, buffersSize: 16384, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.26387467848045 (T) = (0 -651753.422041208) / Math.Sqrt((135780282.1214159 / (5)) + (94767835.81880009 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.1405592179010453 = (571433.2161030801 - 651753.422041208) / 571433.2161030801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 104857600, buffersSize: 65536, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.005161641037911 (T) = (0 -35433206.36752137) / Math.Sqrt((1937953484818.9321 / (6)) + (268931712126.9743 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -0.14547062350428838 = (30933317.39850482 - 35433206.36752137) / 30933317.39850482 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 104857600, buffersSize: 65536, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/9/2022 1:45:57 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.07886549227855 (T) = (0 -26071460.189458687) / Math.Sqrt((485998441090.3823 / (7)) + (9752686589.091682 / (2))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (2) - 2, .025) and -0.15840684839535873 = (22506307.02466343 - 26071460.189458687) / 22506307.02466343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 1048576, buffersSize: 16384, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.553258379825095 (T) = (0 -662105.2457090558) / Math.Sqrt((224611643.0490674 / (5)) + (25141947.59405454 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.17162170976558036 = (565118.6216424162 - 662105.2457090558) / 565118.6216424162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 104857600, buffersSize: 65536, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -4.9418710082776816 (T) = (0 -24935435.531695157) / Math.Sqrt((1925897431326.5474 / (6)) + (295664537870.45404 / (3))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (3) - 2, .025) and -0.14727440428744568 = (21734499.99277389 - 24935435.531695157) / 21734499.99277389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 1048576, buffersSize: 16384, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -6.739825919651657 (T) = (0 -1031340.6570106216) / Math.Sqrt((16557180.54992043 / (5)) + (988043515.3538818 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.11531738806124028 = (924705.9788096771 - 1031340.6570106216) / 924705.9788096771 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor - Duration of single invocation 176.93 μs 194.67 μs 1.10 0.03 False
CtorInvoke - Duration of single invocation 178.76 μs 195.78 μs 1.10 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Ctor(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.331802062428157 (T) = (0 -196037.34676815558) / Math.Sqrt((6704154.162638828 / (5)) + (4650709.637868582 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.10066789195676769 = (178107.6273785368 - 196037.34676815558) / 178107.6273785368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.615402370631644 (T) = (0 -197434.7543503475) / Math.Sqrt((5770741.910318915 / (5)) + (1602459.723035841 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09407781698865315 = (180457.68891811388 - 197434.7543503475) / 180457.68891811388 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 157.67 μs 172.77 μs 1.10 0.03 False
ICollection - Duration of single invocation 27.88 μs 38.59 μs 1.38 0.14 False
ConcurrentBag - Duration of single invocation 150.98 μs 162.76 μs 1.08 0.08 False
SortedList - Duration of single invocation 609.82 μs 1.38 ms 2.26 0.06 False
ImmutableArray - Duration of single invocation 388.91 μs 1.28 ms 3.30 0.05 False
IDictionary - Duration of single invocation 163.39 μs 177.29 μs 1.09 0.03 False
HashSet - Duration of single invocation 139.62 μs 149.27 μs 1.07 0.03 False
List - Duration of single invocation 22.39 μs 33.99 μs 1.52 0.24 False
Array - Duration of single invocation 7.27 μs 7.86 μs 1.08 0.08 False
Queue - Duration of single invocation 40.26 μs 50.08 μs 1.24 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.188767426414454 (T) = (0 -173395.05178963707) / Math.Sqrt((2717081.832046816 / (5)) + (4644185.455094928 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09199401173429742 = (158787.5482158114 - 173395.05178963707) / 158787.5482158114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.092665127436812 (T) = (0 -40575.46384197708) / Math.Sqrt((439316.56375142734 / (5)) + (2362743.468556026 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.44175954675270646 = (28143.017282851164 - 40575.46384197708) / 28143.017282851164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -19.778310091669 (T) = (0 -161673.86433831265) / Math.Sqrt((379389.88239627675 / (5)) + (717266.0040020318 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.06587025443333222 = (151682.49950296833 - 161673.86433831265) / 151682.49950296833 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.406204000164834 (T) = (0 -1401232.50548256) / Math.Sqrt((71770768.09981976 / (5)) + (1062066329.2079369 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.304059536880188 = (608158.115297619 - 1401232.50548256) / 608158.115297619 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -93.8993283240484 (T) = (0 -1272776.2782575197) / Math.Sqrt((46013236.95271356 / (5)) + (311552458.14819443 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.2101505667554457 = (396484.9161402222 - 1272776.2782575197) / 396484.9161402222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.025530833442938 (T) = (0 -177622.801417367) / Math.Sqrt((2027509.4275266042 / (5)) + (4122892.397108655 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.08830104112834639 = (163211.09206438722 - 177622.801417367) / 163211.09206438722 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.40025976690076 (T) = (0 -153347.75938567653) / Math.Sqrt((1060283.8069300107 / (5)) + (8900687.018076548 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09351661845133206 = (140233.5884047668 - 153347.75938567653) / 140233.5884047668 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -25.386811826049627 (T) = (0 -34878.58512156252) / Math.Sqrt((255704.15422373524 / (5)) + (777712.574797364 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.5641903535999663 = (22298.171729092854 - 34878.58512156252) / 22298.171729092854 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/6/2022 2:12:43 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.536255105099054 (T) = (0 -7830.336151341148) / Math.Sqrt((2473.607119847218 / (4)) + (4630.932430820253 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -0.07289211348357831 = (7298.344402883897 - 7830.336151341148) / 7298.344402883897 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.318878322352623 (T) = (0 -52449.39435148142) / Math.Sqrt((576359.2773008771 / (5)) + (2876229.633177402 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.27313222066479564 = (41197.130588756714 - 52449.39435148142) / 41197.130588756714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 13, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Clear - Duration of single invocation 1.02 μs 1.45 μs 1.42 0.14 False
ArrayRetrieve2D - Duration of single invocation 82.31 ms 157.66 ms 1.92 0.10 False
ArrayAssign1D - Duration of single invocation 65.58 ms 107.95 ms 1.65 0.22 False
ArrayAssign3D - Duration of single invocation 88.21 ms 198.51 ms 2.25 0.19 False
ArrayAssign2D - Duration of single invocation 79.27 ms 155.06 ms 1.96 0.19 False
ArrayCopy2D - Duration of single invocation 1.55 μs 3.30 μs 2.13 0.08 False
ArrayRetrieve1D - Duration of single invocation 62.82 ms 103.30 ms 1.64 0.22 False
ArrayCopy3D - Duration of single invocation 1.55 μs 3.29 μs 2.12 0.08 False
ArrayRetrieve3D - Duration of single invocation 103.21 ms 222.71 ms 2.16 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.Clear

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -33.90310618723758 (T) = (0 -1472.5138265582652) / Math.Sqrt((850.4129803141095 / (5)) + (148.27256122534067 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.4956128142611071 = (984.5555029466275 - 1472.5138265582652) / 984.5555029466275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -72.7028528184564 (T) = (0 -159605219.6197883) / Math.Sqrt((383988594681.23627 / (5)) + (4223027197302.944 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.9408731415610823 = (82233720.58794872 - 159605219.6197883) / 82233720.58794872 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign1D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.468364666565565 (T) = (0 -100896717.517239) / Math.Sqrt((4378789822676.818 / (5)) + (28444224454031.14 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.536655833595777 = (65659931.98434065 - 100896717.517239) / 65659931.98434065 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -59.63681378586552 (T) = (0 -202923726.62774724) / Math.Sqrt((5711408244707.631 / (5)) + (10122282092623.89 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.2895178705703656 = (88631641.28838828 - 202923726.62774724) / 88631641.28838828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -31.586121104728004 (T) = (0 -159948755.43818682) / Math.Sqrt((7420131129443.493 / (5)) + (18161999742533.703 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.9406333344245654 = (82420904.86692308 - 159948755.43818682) / 82420904.86692308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -115.04796690871747 (T) = (0 -3287.1577181780413) / Math.Sqrt((112.49515291770578 / (5)) + (812.0151805966383 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.107813117909386 = (1559.5109880701268 - 3287.1577181780413) / 1559.5109880701268 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve1D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.699740792488956 (T) = (0 -101597768.63319597) / Math.Sqrt((3742687907873.9863 / (5)) + (5190673175008.774 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.5331841120118427 = (66265863.204047605 - 101597768.63319597) / 66265863.204047605 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -157.9170188793474 (T) = (0 -3279.043268021261) / Math.Sqrt((21.750880429445193 / (5)) + (456.2162953672598 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.1010222029095238 = (1560.6894888975462 - 3279.043268021261) / 1560.6894888975462 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -49.60802138651579 (T) = (0 -228201026.8065476) / Math.Sqrt((3738923034389.033 / (5)) + (23721416576492.53 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.2819302508881352 = (100003506.55666666 - 228201026.8065476) / 100003506.55666666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 1.16 μs 2.55 μs 2.20 0.05 False
ImmutableSortedDictionary - Duration of single invocation 2.90 ms 3.07 ms 1.06 0.03 False
List - Duration of single invocation 1.13 μs 2.38 μs 2.11 0.03 False
ImmutableList - Duration of single invocation 204.20 μs 360.59 μs 1.77 0.06 False
ConcurrentBag - Duration of single invocation 145.92 μs 154.86 μs 1.06 0.03 False
Queue - Duration of single invocation 1.17 μs 2.49 μs 2.12 0.05 False
ImmutableArray - Duration of single invocation 1.14 μs 1.93 μs 1.69 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.Stack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -46.56928859174266 (T) = (0 -2476.3102735817993) / Math.Sqrt((182.2803367689164 / (5)) + (3097.0634134780926 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.152686971234356 = (1150.3345849498392 - 2476.3102735817993) / 1150.3345849498392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.462806714894165 (T) = (0 -3113776.708308532) / Math.Sqrt((1627472710.4123614 / (5)) + (1214072279.2195613 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.06395392849701488 = (2926608.591696429 - 3113776.708308532) / 2926608.591696429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -61.59652625609406 (T) = (0 -2422.852603824007) / Math.Sqrt((241.47073631043875 / (5)) + (1540.4585170794364 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.1243644318363226 = (1140.5070464909206 - 2422.852603824007) / 1140.5070464909206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -49.53491605661961 (T) = (0 -366293.40932146873) / Math.Sqrt((15605851.955392322 / (5)) + (30279201.764419835 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.7926661653509082 = (204328.84627449198 - 366293.40932146873) / 204328.84627449198 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.542489741741795 (T) = (0 -156497.53502544385) / Math.Sqrt((492300.3255471962 / (5)) + (3671457.476001855 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.06549719392723108 = (146877.4727117038 - 156497.53502544385) / 146877.4727117038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -44.23384559029186 (T) = (0 -2545.487981628783) / Math.Sqrt((66.96305125248597 / (5)) + (3769.4645032214225 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.160886386587127 = (1177.9832560512773 - 2545.487981628783) / 1177.9832560512773 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -71.90480828047107 (T) = (0 -1923.1205159590772) / Math.Sqrt((173.64421206190048 / (5)) + (351.018935118329 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.7059020741660187 = (1127.3334765708942 - 1923.1205159590772) / 1127.3334765708942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObservableCollection - Duration of single invocation 188.90 μs 204.86 μs 1.08 0.03 False
SortedList - Duration of single invocation 594.68 μs 1.36 ms 2.29 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.ObservableCollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.976948235830925 (T) = (0 -206938.17696395627) / Math.Sqrt((469136.90296023706 / (5)) + (10270051.77901827 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09473255195571648 = (189030.8062862163 - 206938.17696395627) / 189030.8062862163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.AddGivenSize&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -71.59003560964116 (T) = (0 -1350269.3434700281) / Math.Sqrt((16470484.140777357 / (5)) + (428412526.1206693 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.257702798440016 = (598072.2282857652 - 1350269.3434700281) / 598072.2282857652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Security.Cryptography.Tests.Perf_CryptoConfig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateFromName - Duration of single invocation 9.90 μs 11.93 μs 1.20 0.08 False
CreateFromName - Duration of single invocation 11.90 μs 13.69 μs 1.15 0.19 False
CreateFromName - Duration of single invocation 8.94 μs 11.19 μs 1.25 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Security.Cryptography.Tests.Perf_CryptoConfig*'

Payloads

Baseline
Compare

Histogram

System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "SHA512")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -49.929730882380305 (T) = (0 -12005.346565797854) / Math.Sqrt((1745.9281009667056 / (5)) + (5986.493117763799 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.2175534208793622 = (9860.221621427623 - 12005.346565797854) / 9860.221621427623 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "RSA")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -5.876337048081003 (T) = (0 -13793.517209755408) / Math.Sqrt((312973.58200373687 / (5)) + (142238.67169562328 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.15402930892968608 = (11952.484311293894 - 13793.517209755408) / 11952.484311293894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "X509Chain")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.26329794431167 (T) = (0 -11033.16234733535) / Math.Sqrt((12445.774635019465 / (5)) + (42948.49229905299 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.2351317690752517 = (8932.781605638664 - 11033.16234733535) / 8932.781605638664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in JetStream.Poker

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Play - Duration of single invocation 413.93 ms 446.47 ms 1.08 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'JetStream.Poker*'

Payloads

Baseline
Compare

Histogram

JetStream.Poker.Play

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.38870753665479 (T) = (0 -455164432.3758242) / Math.Sqrt((6875847299703.623 / (5)) + (33767580647237.03 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.0932310242247534 = (416347891.97333336 - 455164432.3758242) / 416347891.97333336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 13, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CopyTo<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 1.01 μs 2.72 μs 2.71 0.08 False
Array - Duration of single invocation 991.24 ns 2.70 μs 2.72 0.08 False
List - Duration of single invocation 779.69 ns 1.53 μs 1.96 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<Int32>.ImmutableArray(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -145.9030299581224 (T) = (0 -2741.8597662763336) / Math.Sqrt((65.26910265538143 / (5)) + (514.6079092646672 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.7281466500771514 = (1005.026531912716 - 2741.8597662763336) / 1005.026531912716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;Int32&gt;.Array(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -198.661606714997 (T) = (0 -2719.187576267671) / Math.Sqrt((58.858809734012794 / (5)) + (259.970306617586 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.7786258983590277 = (978.6087353009776 - 2719.187576267671) / 978.6087353009776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;Int32&gt;.List(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -261.15664180589573 (T) = (0 -1524.1666457771946) / Math.Sqrt((7.756915066573315 / (5)) + (26.28698823251051 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.9544573358048244 = (779.8413492354691 - 1524.1666457771946) / 779.8413492354691 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 8.19 μs 9.35 μs 1.14 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.893482279576851 (T) = (0 -9613.698997235562) / Math.Sqrt((1884.8510212095057 / (5)) + (40085.99893591364 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.17283991742263097 = (8196.94048132511 - 9613.698997235562) / 8196.94048132511 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Cache

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch_Multithreading - Duration of single invocation 427.06 ms 468.55 ms 1.10 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Cache*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch_Multithreading(total: 40000, unique: 1600, cacheSize: 800)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.937055065914201 (T) = (0 -463911477.9868421) / Math.Sqrt((15584413847520.38 / (5)) + (78786223475078.45 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.08899052123394365 = (426001392.06095237 - 463911477.9868421) / 426001392.06095237 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 13, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 32.34 μs 44.84 μs 1.39 0.22 False
List - Duration of single invocation 26.70 μs 38.01 μs 1.42 0.16 False
Queue - Duration of single invocation 46.94 μs 56.83 μs 1.21 0.03 False
ImmutableArray - Duration of single invocation 478.98 μs 1.40 ms 2.93 0.08 False
SortedList - Duration of single invocation 2.39 ms 3.23 ms 1.35 0.03 False
ConcurrentBag - Duration of single invocation 159.01 μs 170.14 μs 1.07 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.ICollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -20.11803697856907 (T) = (0 -46473.56195876376) / Math.Sqrt((605980.8953659727 / (5)) + (1343399.8131958693 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.4137422854028334 = (32872.72541722237 - 46473.56195876376) / 32872.72541722237 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -16.615094059399134 (T) = (0 -39851.140840383385) / Math.Sqrt((234354.90547253104 / (5)) + (2117303.9030341296 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.46301693989535453 = (27239.01532078899 - 39851.140840383385) / 27239.01532078899 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.595474619121493 (T) = (0 -58726.557706250765) / Math.Sqrt((642924.9078989496 / (5)) + (2612833.6960718688 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.2573781976740504 = (46705.56385889747 - 58726.557706250765) / 46705.56385889747 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -246.67045802432105 (T) = (0 -1407253.4740048363) / Math.Sqrt((9941016.873942915 / (5)) + (48601768.544392735 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.9333776312753745 = (479738.2577002165 - 1407253.4740048363) / 479738.2577002165 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -50.18667062018825 (T) = (0 -3208953.751666667) / Math.Sqrt((833586291.6713498 / (5)) + (435234304.3350784 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.3506228152323847 = (2375906.667261905 - 3208953.751666667) / 2375906.667261905 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.162169673359124 (T) = (0 -169765.26724732597) / Math.Sqrt((3145589.1005152455 / (5)) + (932503.9057326791 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.05885582758709596 = (160328.9728632692 - 169765.26724732597) / 160328.9728632692 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObservableCollection - Duration of single invocation 182.35 μs 198.24 μs 1.09 0.03 False
SortedList - Duration of single invocation 2.30 ms 3.15 ms 1.37 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.ObservableCollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.471237181506687 (T) = (0 -199515.23109825933) / Math.Sqrt((7875378.534118292 / (5)) + (4336771.465769657 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.08390816956048414 = (184070.2346391218 - 199515.23109825933) / 184070.2346391218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.AddGivenSize&lt;String&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -125.06034077511208 (T) = (0 -3144691.241979167) / Math.Sqrt((177806011.696121 / (5)) + (32244002.313460346 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.3562280524753277 = (2318703.875973967 - 3144691.241979167) / 2318703.875973967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayList - Duration of single invocation 2.43 μs 5.19 μs 2.14 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<String>.ArrayList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.88861524851338 (T) = (0 -5015.606085946087) / Math.Sqrt((826.1566224399584 / (5)) + (16537.651104741697 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.0897985043473648 = (2400.042911080769 - 5015.606085946087) / 2400.042911080769 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 248.31 μs 263.69 μs 1.06 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -5.766844382051494 (T) = (0 -267444.93136223766) / Math.Sqrt((8192874.8897675965 / (5)) + (25201744.15762037 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.06468573198726421 = (251196.12607472873 - 267444.93136223766) / 251196.12607472873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Formats.Cbor.Tests.Perf_CborWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteCoseKey - Duration of single invocation 14.94 μs 16.14 μs 1.08 0.11 False
WriteCoseKey - Duration of single invocation 15.48 μs 16.66 μs 1.08 0.08 False
WriteCoseKey - Duration of single invocation 14.38 μs 15.69 μs 1.09 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Cbor.Tests.Perf_CborWriter*'

Payloads

Baseline
Compare

Histogram

System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P384)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -6.842006417617512 (T) = (0 -16718.16807980874) / Math.Sqrt((39052.15537577707 / (5)) + (202788.56425350835 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.10986826377292104 = (15063.20040450249 - 16718.16807980874) / 15063.20040450249 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P521)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.956027672286334 (T) = (0 -16841.626268665874) / Math.Sqrt((28672.15678924121 / (5)) + (23468.335104725567 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09034613656344488 = (15446.128255883352 - 16841.626268665874) / 15446.128255883352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P256)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -14.28028710020091 (T) = (0 -15903.98769924699) / Math.Sqrt((21445.69693895454 / (5)) + (21563.747415949638 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.09690331779163412 = (14498.987687689796 - 15903.98769924699) / 14498.987687689796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 147.84 μs 161.20 μs 1.09 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddDefaultSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<Int32>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -5.65263179112539 (T) = (0 -164819.89079507467) / Math.Sqrt((5277056.340811058 / (5)) + (26789357.799329653 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.10557395293995427 = (149080.83747521712 - 164819.89079507467) / 149080.83747521712 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor - Duration of single invocation 65.14 μs 69.57 μs 1.07 0.03 False
Ctor - Duration of single invocation 39.39 μs 41.81 μs 1.06 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -26.037396164436057 (T) = (0 -69698.92536458248) / Math.Sqrt((78183.32951999747 / (5)) + (61962.415968819725 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.07055903032595298 = (65105.16785175429 - 69698.92536458248) / 65105.16785175429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\\w\\.+-]+@[\\w\\.-]+\\.[\\w\\.-]+", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.475210882382171 (T) = (0 -42066.660169247036) / Math.Sqrt((197548.66849413211 / (5)) + (65747.28792772407 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.06258574163483327 = (39588.95599758914 - 42066.660169247036) / 39588.95599758914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where00ForX - Duration of single invocation 5.53 secs 6.84 secs 1.24 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where00ForX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.413610129310728 (T) = (0 -7092811847.321428) / Math.Sqrt((13809863282526800 / (5)) + (40057051400283136 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.3256153888503851 = (5350580497.916923 - 7092811847.321428) / 5350580497.916923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 25.85 μs 28.87 μs 1.12 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<String, String>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.191641836978731 (T) = (0 -28311.12549999784) / Math.Sqrt((149483.7041566557 / (5)) + (180508.8880686682 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.07478093104560625 = (26341.298661165503 - 28311.12549999784) / 26341.298661165503 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 1.41 secs 7.35 secs 5.21 0.11 False
Test2 - Duration of single invocation 1.35 secs 7.39 secs 5.47 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.51893224265503 (T) = (0 -7212738043.090233) / Math.Sqrt((1329800046436837 / (5)) + (82807497500096350 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -4.360695442297893 = (1345485510.3647618 - 7212738043.090233) / 1345485510.3647618 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -52.66711824047601 (T) = (0 -7416229767.677083) / Math.Sqrt((696487302631043.9 / (5)) + (52553293974435176 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -4.503803418532415 = (1347473593.0257144 - 7416229767.677083) / 1347473593.0257144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Formats.Cbor.Tests.Perf_CborReader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SkipValue - Duration of single invocation 16.51 μs 18.22 μs 1.10 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Cbor.Tests.Perf_CborReader*'

Payloads

Baseline
Compare

Histogram

System.Formats.Cbor.Tests.Perf_CborReader.SkipValue(encoding: (Map, Strict))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.23082816205263 (T) = (0 -18381.172062465605) / Math.Sqrt((109113.19495859309 / (5)) + (19577.760550285017 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.10008229440625181 = (16708.90637539666 - 18381.172062465605) / 16708.90637539666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 13, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToArray - Duration of single invocation 11.99 μs 19.03 μs 1.59 0.08 False
TakeLastHalf - Duration of single invocation 27.03 μs 32.47 μs 1.20 0.03 False
Except - Duration of single invocation 57.24 μs 65.50 μs 1.14 0.03 False
ToDictionary - Duration of single invocation 37.23 μs 45.79 μs 1.23 0.06 False
ToList - Duration of single invocation 12.33 μs 20.67 μs 1.68 0.11 False
SelectToList - Duration of single invocation 14.04 μs 22.43 μs 1.60 0.03 False
ToArray - Duration of single invocation 472.90 ns 1.75 μs 3.71 0.03 False
ToList - Duration of single invocation 503.64 ns 1.78 μs 3.53 0.05 False
Intersect - Duration of single invocation 65.47 μs 73.60 μs 1.12 0.03 False
SelectToArray - Duration of single invocation 14.13 μs 21.25 μs 1.50 0.08 False
Distinct - Duration of single invocation 37.66 μs 43.71 μs 1.16 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.ToArray(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -46.16395064157009 (T) = (0 -19367.63557682959) / Math.Sqrt((28126.467126880783 / (5)) + (74648.11244781812 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.590997560405362 = (12173.265414621384 - 19367.63557682959) / 12173.265414621384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.TakeLastHalf(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.335571782348294 (T) = (0 -33443.0019597554) / Math.Sqrt((110781.46642717694 / (5)) + (591206.0931032836 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.2331141075012638 = (27120.768269793818 - 33443.0019597554) / 27120.768269793818 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Except(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.140947089326513 (T) = (0 -66837.50973789823) / Math.Sqrt((1035291.8244551299 / (5)) + (1222590.3348003144 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.14951356020352863 = (58144.16815236545 - 66837.50973789823) / 58144.16815236545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToDictionary(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -23.722799736707934 (T) = (0 -46761.17841948964) / Math.Sqrt((111673.35186501162 / (5)) + (558638.8832294857 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.25657838661465604 = (37213.10100316844 - 46761.17841948964) / 37213.10100316844 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToList(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -31.593115588281357 (T) = (0 -21091.82509575465) / Math.Sqrt((37510.452863727805 / (5)) + (296467.3390970783 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.7480408309712725 = (12065.97965107903 - 21091.82509575465) / 12065.97965107903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -35.41636299646669 (T) = (0 -23110.204084870657) / Math.Sqrt((38436.24542904191 / (5)) + (221814.72073951364 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.6262388256959884 = (14210.830364955826 - 23110.204084870657) / 14210.830364955826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -134.7148533617312 (T) = (0 -1733.0428162247244) / Math.Sqrt((86.16619481840172 / (5)) + (277.05020923345165 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.609357369473156 = (480.15273601951196 - 1733.0428162247244) / 480.15273601951196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToList(input: ICollection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -89.85357838210535 (T) = (0 -1779.2600958740186) / Math.Sqrt((50.8939852359143 / (5)) + (757.6668290051338 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.4899478950371696 = (509.8242579507247 - 1779.2600958740186) / 509.8242579507247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Intersect(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.64040598711943 (T) = (0 -74742.78286767627) / Math.Sqrt((432080.84227396076 / (5)) + (1274829.9665578294 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.13142487049930937 = (66060.75649962646 - 74742.78286767627) / 66060.75649962646 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -28.20206927444973 (T) = (0 -21898.12747728255) / Math.Sqrt((56195.235855048675 / (5)) + (249093.94588752356 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.5365332736328478 = (14251.64547560268 - 21898.12747728255) / 14251.64547560268 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Distinct(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -17.715246343927735 (T) = (0 -44724.83185923525) / Math.Sqrt((46430.121083886195 / (5)) + (568192.1345441522 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.18215507176174664 = (37833.30370742525 - 44724.83185923525) / 37833.30370742525 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 206.81 μs 375.08 μs 1.81 0.11 False
ImmutableSortedDictionary - Duration of single invocation 1.22 ms 1.38 ms 1.13 0.03 False
List - Duration of single invocation 751.66 ns 2.04 μs 2.72 0.11 False
ImmutableArray - Duration of single invocation 756.93 ns 1.56 μs 2.06 0.03 False
Queue - Duration of single invocation 817.69 ns 2.09 μs 2.55 0.11 False
Stack - Duration of single invocation 798.66 ns 2.10 μs 2.63 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -58.588503265389186 (T) = (0 -379830.43176942656) / Math.Sqrt((3045807.954077088 / (5)) + (31843519.087220732 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.8233488135772244 = (208314.73876040097 - 379830.43176942656) / 208314.73876040097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -22.474500368291864 (T) = (0 -1397371.6642609127) / Math.Sqrt((171931940.46094424 / (5)) + (82613667.63720997 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.13548738838228966 = (1230636.0057875458 - 1397371.6642609127) / 1230636.0057875458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -99.46721463473361 (T) = (0 -2071.160008624205) / Math.Sqrt((19.335331299830905 / (5)) + (689.1114408589442 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.7577406327397553 = (751.0350988180323 - 2071.160008624205) / 751.0350988180323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -170.31749366344255 (T) = (0 -1557.1732060638626) / Math.Sqrt((54.60490382213 / (5)) + (44.4516025207586 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.0551358247558131 = (757.6984388605472 - 1557.1732060638626) / 757.6984388605472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.127679663175414 (T) = (0 -2101.8381619564366) / Math.Sqrt((159.57776853405394 / (5)) + (5081.687481612972 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.6336658350128423 = (798.0656216950118 - 2101.8381619564366) / 798.0656216950118 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -74.57646498641272 (T) = (0 -2112.713120370712) / Math.Sqrt((197.93000597618848 / (5)) + (1096.2637728017369 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.667744215905715 = (791.9474092659343 - 2112.713120370712) / 791.9474092659343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayIntArrayCtor - Duration of single invocation 859.25 ns 2.58 μs 3.01 0.05 False
BitArrayIntArrayCtor - Duration of single invocation 498.27 ns 2.37 μs 4.76 0.05 False
BitArrayCopyToIntArray - Duration of single invocation 250.21 ns 1.00 μs 4.00 0.03 False
BitArrayCopyToIntArray - Duration of single invocation 276.28 ns 1.03 μs 3.72 0.14 False
BitArrayBitArrayCtor - Duration of single invocation 508.98 ns 2.25 μs 4.43 0.03 False
BitArrayBitArrayCtor - Duration of single invocation 508.94 ns 2.24 μs 4.40 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -62.12288580900768 (T) = (0 -2619.500956352268) / Math.Sqrt((163.35317544425476 / (5)) + (3062.6720230331125 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.0310375447560736 = (864.2258360950378 - 2619.500956352268) / 864.2258360950378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -46.67247246636903 (T) = (0 -2270.040498746657) / Math.Sqrt((10.368455763624832 / (5)) + (5723.520585175208 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -3.510463999175991 = (503.2831431891191 - 2270.040498746657) / 503.2831431891191 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToIntArray(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -76.1522438587282 (T) = (0 -986.9962136226212) / Math.Sqrt((10.591956447068169 / (5)) + (364.5412667868222 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.922729791659573 = (251.6095336775814 - 986.9962136226212) / 251.6095336775814 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToIntArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -26.26093736576531 (T) = (0 -1047.4656151171407) / Math.Sqrt((2.6727078135218076 / (5)) + (3446.483268501693 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.7899566884255114 = (276.37931016892355 - 1047.4656151171407) / 276.37931016892355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -102.2094066176994 (T) = (0 -2269.202301100881) / Math.Sqrt((85.36611074115176 / (5)) + (1111.5501595579715 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -3.4163813437230117 = (513.8148462487486 - 2269.202301100881) / 513.8148462487486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -98.65314638456874 (T) = (0 -2231.2652287280116) / Math.Sqrt((7.848153806912445 / (5)) + (1217.8265092664874 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -3.41427418653121 = (505.4659349290142 - 2231.2652287280116) / 505.4659349290142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 4.25 ms 5.84 ms 1.37 0.03 False
List - Duration of single invocation 288.00 μs 689.54 μs 2.39 0.03 False
Queue - Duration of single invocation 83.03 μs 91.04 μs 1.10 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -41.53893928894691 (T) = (0 -5917073.4613591265) / Math.Sqrt((1817484332.9825141 / (5)) + (5259758297.70226 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.40372145431563355 = (4215276.074300596 - 5917073.4613591265) / 4215276.074300596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -165.50102656761905 (T) = (0 -683686.9807914401) / Math.Sqrt((3889996.169352291 / (5)) + (19638411.6097236 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -1.3657918568262057 = (288988.6440426888 - 683686.9807914401) / 288988.6440426888 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -5.39902870947945 (T) = (0 -96254.10668358502) / Math.Sqrt((2114806.2306864946 / (5)) + (16649665.03037709 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -0.13650739642740783 = (84692.89947972025 - 96254.10668358502) / 84692.89947972025 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline b399eed41e575b5d03d0901e9d4776aa9a93b9a9
Compare c1576ffb746b8ffc5bbfb931e1bd6cd0b43727eb
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayList - Duration of single invocation 176.33 μs 575.10 μs 3.26 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<Int32>.ArrayList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.099797087215727 (T) = (0 -598739.490933642) / Math.Sqrt((16430679.918571841 / (5)) + (1225898242.498478 / (4))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (4) - 2, .025) and -2.4295487721714535 = (174582.58526370046 - 598739.490933642) / 174582.58526370046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

Same as dotnet/runtime#79607. Closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos