Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 6/22/2022 10:05:16 PM #6389

Closed
performanceautofiler bot opened this issue Jun 28, 2022 · 1 comment
Closed

[Perf] Changes at 6/22/2022 10:05:16 PM #6389

performanceautofiler bot opened this issue Jun 28, 2022 · 1 comment
Assignees
@dakersnar
Labels
arch-x86 branch-refs/heads/main kind-micro Noise os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jun 28, 2022 •
edited
Loading

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare 70df1f53ef459154c1bcc109848980fd79be5c63
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 2.35 μs 2.58 μs 1.10 0.10 False
IEnumerable - Duration of single invocation 2.42 μs 3.05 μs 1.26 0.03 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.578165178079171 > 2.3430591400194665.
IsChangePoint: Marked as a change because one of 5/1/2022 2:52:05 AM, 5/1/2022 6:14:18 PM, 5/15/2022 3:27:46 PM, 5/23/2022 3:43:39 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -12.133721311204491 (T) = (0 -2567.791328612082) / Math.Sqrt((18365.729039961934 / (39)) + (109.1810743475931 / (19))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (19) - 2, .025) and -0.1150344933960434 = (2302.8806228149942 - 2567.791328612082) / 2302.8806228149942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.IEnumerable(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0543423879304714 > 2.5447089602576156.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/15/2022 3:27:46 PM, 5/23/2022 3:43:39 PM, 5/28/2022 5:31:50 AM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.38372930309004 (T) = (0 -3056.2235698014533) / Math.Sqrt((53920.33575768455 / (39)) + (1.9833218977734604 / (21))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (21) - 2, .025) and -0.2121254482179918 = (2521.375633433458 - 3056.2235698014533) / 2521.375633433458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x86
OS Windows 10.0.18362
Baseline 9b4f152802b6305438bc1f6ddf4733104bb57f6e
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NewCustomConverter - Duration of single invocation 24.43 μs 28.81 μs 1.18 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.808093575418997 > 25.941618380497726.
IsChangePoint: Marked as a change because one of 4/26/2022 9:50:05 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -29.156791001618803 (T) = (0 -30253.389403379395) / Math.Sqrt((541596.242743947 / (36)) + (448254.3398212604 / (20))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (20) - 2, .025) and -0.22929113394580106 = (24610.43488231426 - 30253.389403379395) / 24610.43488231426 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InstanceDelegate - Duration of single invocation 218.30 μs 249.48 μs 1.14 0.03 False
StaticDelegate - Duration of single invocation 280.66 μs 343.01 μs 1.22 0.02 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.InstanceDelegate

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 249.47883597883597 > 229.21518500434024.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/15/2022 3:27:46 PM, 5/28/2022 5:31:50 AM, 5/31/2022 6:06:22 PM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.385303851534584 (T) = (0 -249470.69669950122) / Math.Sqrt((128909802.57850255 / (39)) + (19.71003328393738 / (18))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (18) - 2, .025) and -0.11711367462862894 = (223317.19892555676 - 249470.69669950122) / 223317.19892555676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.StaticDelegate

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 343.01223871237465 > 294.6893303571429.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/15/2022 3:27:46 PM, 5/23/2022 3:43:39 PM, 5/28/2022 5:31:50 AM, 5/31/2022 9:21:33 PM, 6/20/2022 7:45:04 PM, 6/21/2022 12:11:10 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.047824359259293 (T) = (0 -343024.57024821505) / Math.Sqrt((531122852.9913825 / (38)) + (203.33902289303433 / (18))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (18) - 2, .025) and -0.18078408824009487 = (290505.7526304217 - 343024.57024821505) / 290505.7526304217 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObjFooIsObj - Duration of single invocation 311.84 μs 440.93 μs 1.41 0.07 False
FooObjIsFoo - Duration of single invocation 438.18 μs 561.32 μs 1.28 0.07 False
IFooFooIsIFoo - Duration of single invocation 311.82 μs 440.00 μs 1.41 0.06 False
FooObjIsNull - Duration of single invocation 311.87 μs 440.13 μs 1.41 0.06 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 440.92663194444447 > 327.45213305995475.
IsChangePoint: Marked as a change because one of 5/20/2022 5:23:40 AM, 5/23/2022 3:43:39 PM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.435567116718525 (T) = (0 -440547.3769427909) / Math.Sqrt((2214564900.1482 / (39)) + (91338.25867976341 / (20))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (20) - 2, .025) and -0.32789453700404986 = (331763.82963118353 - 440547.3769427909) / 331763.82963118353 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.FooObjIsFoo

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 561.3209821428571 > 460.20939756944443.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 5/15/2022 3:27:46 PM, 5/31/2022 9:21:33 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.456646423085415 (T) = (0 -561333.4658719256) / Math.Sqrt((2005481010.258049 / (39)) + (8821.950779521156 / (19))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (19) - 2, .025) and -0.22651593128223188 = (457665.0425446108 - 561333.4658719256) / 457665.0425446108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 439.99911954365075 > 327.4495119485294.
IsChangePoint: Marked as a change because one of 5/20/2022 5:23:40 AM, 5/23/2022 3:43:39 PM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.461417183916739 (T) = (0 -440393.7922421906) / Math.Sqrt((2203140737.924957 / (39)) + (102544.12620559028 / (20))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (20) - 2, .025) and -0.3277017080651597 = (331696.3362832229 - 440393.7922421906) / 331696.3362832229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 440.12783119658127 > 327.44287283182507.
IsChangePoint: Marked as a change because one of 5/20/2022 5:23:40 AM, 5/23/2022 3:43:39 PM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -14.49595426445704 (T) = (0 -440427.2799987161) / Math.Sqrt((2193388193.093792 / (39)) + (880206.5106772278 / (21))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (21) - 2, .025) and -0.327883732079724 = (331676.08681290294 - 440427.2799987161) / 331676.08681290294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in BenchmarksGame.ReverseComplement_1

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 479.00 μs 532.49 μs 1.11 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'BenchmarksGame.ReverseComplement_1*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.ReverseComplement_1.RunBench

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 532.4871264367818 > 502.5755724431819.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/11/2022 9:09:01 PM, 5/15/2022 3:27:46 PM, 6/2/2022 5:58:32 PM, 6/17/2022 12:17:51 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -15.734599222225155 (T) = (0 -529546.9013307334) / Math.Sqrt((256940015.0769635 / (39)) + (22339579.823401343 / (20))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (20) - 2, .025) and -0.08989317247879301 = (485870.4639156158 - 529546.9013307334) / 485870.4639156158 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ctor_seeded - Duration of single invocation 435.97 ns 491.20 ns 1.13 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.ctor_seeded

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 491.20252652750514 > 458.5444042033312.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 6/10/2022 11:27:32 AM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -84.22084300238633 (T) = (0 -490.90477637658114) / Math.Sqrt((14.295957077417432 / (39)) + (0.8210116392768012 / (20))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (20) - 2, .025) and -0.12300677335724966 = (437.13429698114123 - 490.90477637658114) / 437.13429698114123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 39.94 μs 42.48 μs 1.06 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.483898243400624 > 41.320500055100055.
IsChangePoint: Marked as a change because one of 5/15/2022 3:27:46 PM, 6/8/2022 9:59:38 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -11.37347714159257 (T) = (0 -42705.62104249083) / Math.Sqrt((1539735.66590622 / (39)) + (307474.61896685837 / (20))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (20) - 2, .025) and -0.06652484482200517 = (40041.843609951786 - 42705.62104249083) / 40041.843609951786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d3f7e0145437ca3afe7d8a0bab38cc95040cfc28
Compare d17741d0c44b13cb4df864e0184cf1e499edcd57
Diff Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentStack - Duration of single invocation 5.26 μs 5.88 μs 1.12 0.04 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.ConcurrentStack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.881984082397004 > 5.546923702999873.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/15/2022 3:27:46 PM, 5/24/2022 3:42:52 PM, 5/28/2022 5:31:50 AM, 6/7/2022 6:20:49 PM, 6/22/2022 6:53:40 PM, 6/28/2022 4:38:49 AM falls between 6/19/2022 1:08:59 PM and 6/28/2022 4:38:49 AM.
IsRegressionStdDev: Marked as regression because -40.63745593602775 (T) = (0 -5848.082538095609) / Math.Sqrt((4405.60868739138 / (38)) + (1121.4514198458605 / (18))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (18) - 2, .025) and -0.10225838451058146 = (5305.545977490788 - 5848.082538095609) / 5305.545977490788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakersnar dakersnar

Labels
arch-x86 branch-refs/heads/main kind-micro Noise os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dakersnar