Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 5/30/2022 5:18:40 PM #70032

Closed
performanceautofiler bot opened this issue May 31, 2022 · 8 comments
Closed

[Perf] Changes at 5/30/2022 5:18:40 PM #70032

performanceautofiler bot opened this issue May 31, 2022 · 8 comments
Assignees
@AndyAyersMS
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 14.37 ns 15.55 ns 1.08 0.11 False
TryParseSpan - Duration of single invocation 20.53 ns 22.18 ns 1.08 0.13 False
Parse - Duration of single invocation 20.47 ns 22.81 ns 1.11 0.08 False
TryParseSpan - Duration of single invocation 13.88 ns 15.60 ns 1.12 0.22 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.Parse(value: "12345")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.5548743923072 > 15.104996994035885.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -13.40426032398642 (T) = (0 -15.56223845074927) / Math.Sqrt((0.3997461793558516 / (60)) + (4.075479048722243E-05 / (3))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (3) - 2, .025) and -0.07570467503622394 = (14.467017585681885 - 15.56223845074927) / 14.467017585681885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.1770148518681 > 21.541607506950193.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -10.80960030338378 (T) = (0 -22.56093550737766) / Math.Sqrt((0.2787473236158091 / (62)) + (0.11560187936158357 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11035710442507184 = (20.318630301428488 - 22.56093550737766) / 20.318630301428488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.Parse(value: "2147483647")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.814647795298296 > 21.486482650418488.
IsChangePoint: Marked as a change because one of 4/2/2022 7:01:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -10.171669234588604 (T) = (0 -22.45151356450916) / Math.Sqrt((0.2517145922125223 / (62)) + (0.12021022290261323 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.1051835748492809 = (20.31473691379369 - 22.45151356450916) / 20.31473691379369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParseSpan(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.596814910317187 > 14.573614871516464.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -14.607255018359659 (T) = (0 -15.744000886858773) / Math.Sqrt((0.4095120601365061 / (62)) + (0.016280183439957862 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11330011734528359 = (14.141740076701943 - 15.744000886858773) / 14.141740076701943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 20.19 ns 24.35 ns 1.21 0.13 False
TryParse - Duration of single invocation 14.69 ns 16.61 ns 1.13 0.11 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.TryParse(value: "4294967295")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.350924724827788 > 20.77327058188295.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -22.410759386694327 (T) = (0 -24.03120853435588) / Math.Sqrt((0.6039468105066214 / (58)) + (0.07850560183156165 / (3))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (58) + (3) - 2, .025) and -0.21708638417156606 = (19.74486679572313 - 24.03120853435588) / 19.74486679572313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.611123351515296 > 15.428355856989672.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -17.621587221432215 (T) = (0 -16.721989773685586) / Math.Sqrt((0.3029141960779122 / (60)) + (0.03013464880684888 / (3))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (3) - 2, .025) and -0.14871849638918638 = (14.557082371571884 - 16.721989773685586) / 14.557082371571884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetFileName - Duration of single invocation 30.10 ns 33.15 ns 1.10 0.19 False
GetFileNameWithoutExtension - Duration of single invocation 32.28 ns 36.29 ns 1.12 0.15 False
GetDirectoryName - Duration of single invocation 52.35 ns 58.63 ns 1.12 0.13 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetFileName

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.14910856830112 > 31.725914434803297.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -18.711211553238513 (T) = (0 -32.89212759051452) / Math.Sqrt((0.25013250022765565 / (61)) + (0.04966770012052775 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.0890391706491966 = (30.202887533335378 - 32.89212759051452) / 30.202887533335378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetFileNameWithoutExtension

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.289107835551704 > 34.315962475343746.
IsChangePoint: Marked as a change because one of 5/11/2022 3:53:15 AM, 5/17/2022 4:22:41 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.72910225560168 (T) = (0 -36.48132903744981) / Math.Sqrt((0.4577655435142495 / (61)) + (0.0413827433931418 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.10488230228654973 = (33.01829431239132 - 36.48132903744981) / 33.01829431239132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetDirectoryName

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.63079903237569 > 55.01537859752237.
IsChangePoint: Marked as a change because one of 4/22/2022 2:26:33 AM, 4/22/2022 1:40:09 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -33.69073916010231 (T) = (0 -58.60874812444556) / Math.Sqrt((2.085879252477213 / (61)) + (0.0036670744211391624 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.12130489025877728 = (52.268342565526225 - 58.60874812444556) / 52.268342565526225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 4.34 μs 5.31 μs 1.22 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.3097522369464265 > 4.576847852214168.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -9.341961957849813 (T) = (0 -5168.539934723053) / Math.Sqrt((44426.56451919794 / (62)) + (15288.162116411604 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.1598265452989286 = (4456.304225552051 - 5168.539934723053) / 4456.304225552051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 118.15 ns 149.08 ns 1.26 0.01 False
TryParse - Duration of single invocation 113.02 ns 145.46 ns 1.29 0.04 False
Parse - Duration of single invocation 116.84 ns 147.15 ns 1.26 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 149.07886804264805 > 123.65873408194824.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -176.30090851447954 (T) = (0 -149.0854426817671) / Math.Sqrt((1.9879961140862232 / (61)) + (0.001453889677068605 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.2739890133379836 = (117.02254974016435 - 149.0854426817671) / 117.02254974016435 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 145.46473113096087 > 119.01973480938754.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -101.30717916952578 (T) = (0 -145.44026233215502) / Math.Sqrt((5.715957629479416 / (62)) + (0.0029064375805444706 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.27001119658342354 = (114.51888197790502 - 145.44026233215502) / 114.51888197790502 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.Parse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.14664377602486 > 117.53474394091785.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -91.93617956288767 (T) = (0 -146.71752677361312) / Math.Sqrt((5.061785415961881 / (62)) + (0.15736431344164967 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.29779662414423186 = (113.05124704755552 - 146.71752677361312) / 113.05124704755552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline b8acc97aba7662204c2a6de8f6090fc115a41c9f
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterUInt64 - Duration of single invocation 6.33 ns 7.72 ns 1.22 0.01 False
FormatterUInt32 - Duration of single invocation 6.29 ns 7.48 ns 1.19 0.03 False
FormatterUInt32 - Duration of single invocation 13.03 ns 14.26 ns 1.09 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.716702027898865 > 6.645223423740484.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 5:23:46 PM, 5/23/2022 3:43:39 PM, 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -32.598177930624075 (T) = (0 -7.72627851960429) / Math.Sqrt((0.05569286433136878 / (61)) + (0.00023619372074365139 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.1532265847236033 = (6.699705523573294 - 7.72627851960429) / 6.699705523573294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.478123254655358 > 6.602982892192463.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 3:10:19 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -37.1276609755036 (T) = (0 -7.475300181018473) / Math.Sqrt((0.03365526257398537 / (62)) + (0.0001603703573483267 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.13802117801209834 = (6.568682837762623 - 7.475300181018473) / 6.568682837762623 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.262508361901494 > 13.65358935171701.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 5:23:46 PM, 5/23/2022 3:43:39 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -36.325552219793174 (T) = (0 -13.877533769712803) / Math.Sqrt((0.0016081107175497978 / (21)) + (0.02042657686046091 / (43))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (43) - 2, .025) and -0.06550473743110073 = (13.024375474079179 - 13.877533769712803) / 13.024375474079179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 14.25 ns 16.68 ns 1.17 0.16 False
ToString - Duration of single invocation 11.53 ns 13.97 ns 1.21 0.07 False
TryParse - Duration of single invocation 14.90 ns 16.99 ns 1.14 0.14 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.TryParse(value: "32767")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.684172177665495 > 15.264280202338327.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -36.017600746595534 (T) = (0 -16.654300462968283) / Math.Sqrt((0.145794555338259 / (59)) + (0.0008177345806941237 / (3))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (59) + (3) - 2, .025) and -0.12775213454657308 = (14.767695801936437 - 16.654300462968283) / 14.767695801936437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.ToString(value: 32767)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.970511212518261 > 12.113592516296148.
IsChangePoint: Marked as a change because one of 4/15/2022 9:40:04 PM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -54.89665480793696 (T) = (0 -13.96947092043421) / Math.Sqrt((0.1246689760816977 / (61)) + (0.00044405825897137147 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.22545577531189687 = (11.399408450197862 - 13.96947092043421) / 11.399408450197862 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.TryParse(value: "-32768")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.99456162230395 > 15.689058377667552.
IsChangePoint: Marked as a change because one of 5/16/2022 12:27:40 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -21.917868267261568 (T) = (0 -17.00492699841027) / Math.Sqrt((0.37046122313978247 / (62)) + (8.140462336419778E-05 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11093585876301462 = (15.306848603613009 - 17.00492699841027) / 15.306848603613009 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AppendLine_Strings - Duration of single invocation 224.24 ns 236.50 ns 1.05 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.AppendLine_Strings

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 236.50424263494025 > 236.05565004096457.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -45.86167105878677 (T) = (0 -236.69183426773847) / Math.Sqrt((2.6037737499309066 / (62)) + (0.09574735822032239 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.055606631680901235 = (224.22351959919092 - 236.69183426773847) / 224.22351959919092 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 13.91 μs 15.71 μs 1.13 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<String, String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.712704545454546 > 14.648930340530795.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.26780222645648 (T) = (0 -15670.076459769529) / Math.Sqrt((27561.18341781106 / (62)) + (12418.07817359439 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11176749929286844 = (14094.74235371727 - 15670.076459769529) / 14094.74235371727 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 29.37 μs 37.77 μs 1.29 0.32 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.Queue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.77271717767296 > 30.829469983552634.
IsChangePoint: Marked as a change because one of 4/7/2022 6:53:49 PM, 4/22/2022 6:17:37 AM, 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -24.29530985938782 (T) = (0 -37444.77588618398) / Math.Sqrt((299357.58856600476 / (60)) + (329399.73843577947 / (3))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (3) - 2, .025) and -0.28176757444056805 = (29213.389878837344 - 37444.77588618398) / 29213.389878837344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 14.52 ns 16.25 ns 1.12 0.09 False
Parse - Duration of single invocation 14.25 ns 15.98 ns 1.12 0.11 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.TryParse(value: "12345")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.245205049826843 > 15.159886688670834.
IsChangePoint: Marked as a change because one of 3/31/2022 1:23:38 AM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -17.498063248327504 (T) = (0 -16.25560113308216) / Math.Sqrt((0.2928925884971101 / (62)) + (0.018888689950187307 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.12739746583376968 = (14.418695824422745 - 16.25560113308216) / 14.418695824422745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt16.Parse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.983291559227922 > 14.887861699367225.
IsChangePoint: Marked as a change because one of 4/8/2022 10:19:59 AM, 4/13/2022 5:30:39 PM, 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -33.52666464372722 (T) = (0 -15.971841712510253) / Math.Sqrt((0.11548116549693024 / (61)) + (0.0017950295662565782 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.11703946121394744 = (14.298368380963717 - 15.971841712510253) / 14.298368380963717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline b8acc97aba7662204c2a6de8f6090fc115a41c9f
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 20.63 μs 24.31 μs 1.18 0.02 False
Dictionary - Duration of single invocation 19.42 μs 22.34 μs 1.15 0.04 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.313903100775196 > 21.60848830837679.
IsChangePoint: Marked as a change because one of 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -107.98804464896342 (T) = (0 -24265.717672529918) / Math.Sqrt((32516.27056179387 / (61)) + (1813.4704420140454 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.1766022672458173 = (20623.551686102855 - 24265.717672529918) / 20623.551686102855 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.337953443877545 > 20.264818790683897.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -123.2961129210471 (T) = (0 -22332.508912467078) / Math.Sqrt((33660.65732496015 / (61)) + (317.38325609791616 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.16492811536942936 = (19170.718448481133 - 22332.508912467078) / 19170.718448481133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 18.39 μs 21.54 μs 1.17 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryAddDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.54344117024654 > 19.551999397248352.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -25.83603892499543 (T) = (0 -21681.57285141553) / Math.Sqrt((27006.319792869188 / (62)) + (42199.03934403939 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.16754060325266487 = (18570.294507114002 - 21681.57285141553) / 18570.294507114002 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Hashtable - Duration of single invocation 35.39 μs 38.16 μs 1.08 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<String>.Hashtable(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.16276699029127 > 37.2270942449095.
IsChangePoint: Marked as a change because one of 4/16/2022 10:12:41 PM, 5/11/2022 3:41:15 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -5.242988912412349 (T) = (0 -38646.710004648376) / Math.Sqrt((271016.49724741187 / (62)) + (1030359.6301488609 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.08696881336388426 = (35554.57114270548 - 38646.710004648376) / 35554.57114270548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 89.50 ns 97.75 ns 1.09 0.03 False
TryParse - Duration of single invocation 94.68 ns 105.38 ns 1.11 0.02 False
TryParse - Duration of single invocation 93.39 ns 102.37 ns 1.10 0.01 False
TryParse - Duration of single invocation 55.51 ns 65.03 ns 1.17 0.21 False
Parse - Duration of single invocation 55.72 ns 63.68 ns 1.14 0.17 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Single*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Single.Parse(value: "3.4028235E+38")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 97.7452111145251 > 94.32677585644704.
IsChangePoint: Marked as a change because one of 4/25/2022 5:27:54 PM, 5/9/2022 5:23:46 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -32.86130092934426 (T) = (0 -97.7965635316694) / Math.Sqrt((1.8093273453016676 / (62)) + (0.006552617523154097 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.06327661749425861 = (91.97659566909218 - 97.7965635316694) / 91.97659566909218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Single.TryParse(value: "-3.4028235E+38")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 105.37988671394405 > 99.5281671482242.
IsChangePoint: Marked as a change because one of 4/22/2022 2:26:33 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -38.041901602253326 (T) = (0 -105.50849160717405) / Math.Sqrt((3.600644028013107 / (62)) + (0.02440770662074248 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.10226454202200613 = (95.71975472750681 - 105.50849160717405) / 95.71975472750681 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Single.TryParse(value: "3.4028235E+38")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 102.37381125121198 > 96.89056759490991.
IsChangePoint: Marked as a change because one of 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -44.274506693589466 (T) = (0 -102.47290046299928) / Math.Sqrt((2.8331247839214586 / (61)) + (0.013322177545051208 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.10798822966844081 = (92.48554968283707 - 102.47290046299928) / 92.48554968283707 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Single.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.03154453635959 > 60.15832686910174.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.412297618597655 (T) = (0 -65.05969258936976) / Math.Sqrt((5.713343964896919 / (61)) + (0.00498409850807479 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.12499119410178804 = (57.83129052962456 - 65.05969258936976) / 57.83129052962456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Single.Parse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.678957114425295 > 58.486231052218436.
IsChangePoint: Marked as a change because one of 4/14/2022 8:50:18 PM, 5/23/2022 1:47:22 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -13.93748667187527 (T) = (0 -64.13263934017586) / Math.Sqrt((6.349744251529737 / (62)) + (0.37449493200594075 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11557178827479832 = (57.48858120493998 - 64.13263934017586) / 57.48858120493998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Replace_String - Duration of single invocation 10.43 ns 11.75 ns 1.13 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.749910611541049 > 10.951987119898856.
IsChangePoint: Marked as a change because one of 4/15/2022 9:40:04 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -27.53800848856966 (T) = (0 -11.749645523596579) / Math.Sqrt((0.1339923586892671 / (62)) + (0.0009149523597719594 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.13172167617917274 = (10.38209815267017 - 11.749645523596579) / 10.38209815267017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 13.99 μs 16.08 μs 1.15 0.01 False
IDictionary - Duration of single invocation 15.18 μs 18.53 μs 1.22 0.00 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.079949215312407 > 14.655861084182071.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -33.73975613206283 (T) = (0 -16028.224020448193) / Math.Sqrt((49819.33905175898 / (59)) + (10071.001227440074 / (3))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (59) + (3) - 2, .025) and -0.15800202894473123 = (13841.27455722548 - 16028.224020448193) / 13841.27455722548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.AddGivenSize&lt;String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.52659515295987 > 15.975216572215668.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -52.09700471208975 (T) = (0 -18462.856408152165) / Math.Sqrt((21178.39404359819 / (58)) + (10683.341917529957 / (3))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (58) + (3) - 2, .025) and -0.2147842734757905 = (15198.465119510878 - 18462.856408152165) / 15198.465119510878 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 9.12 ns 10.83 ns 1.19 0.08 False
TryParse - Duration of single invocation 12.42 ns 13.74 ns 1.11 0.15 False
TryParse - Duration of single invocation 13.20 ns 14.43 ns 1.09 0.19 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.ToString(value: 127)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.82575606694572 > 9.605906673560924.
IsChangePoint: Marked as a change because one of 4/15/2022 9:40:04 PM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -30.672204361488262 (T) = (0 -10.827155552995182) / Math.Sqrt((0.19129848706748911 / (62)) + (5.0791335322808856E-05 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.18735197811686405 = (9.118741327375401 - 10.827155552995182) / 9.118741327375401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_SByte.TryParse(value: "127")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.739213575995837 > 13.060641543274961.
IsChangePoint: Marked as a change because one of 4/22/2022 6:17:37 AM, 4/25/2022 5:27:54 PM, 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -29.334662771643025 (T) = (0 -13.789647009781092) / Math.Sqrt((0.06969854976826341 / (60)) + (0.0021535605684264484 / (4))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (4) - 2, .025) and -0.09614448649137747 = (12.58013626827613 - 13.789647009781092) / 12.58013626827613 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_SByte.TryParse(value: "-128")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.434525139248803 > 13.831491199759604.
IsChangePoint: Marked as a change because one of 4/8/2022 8:59:46 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -4.7426580989113205 (T) = (0 -14.71643466909857) / Math.Sqrt((0.2619187127576631 / (62)) + (0.2313195786065611 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.1012084719733447 = (13.363895251120796 - 14.71643466909857) / 13.363895251120796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Order00ManualX - Duration of single invocation 121.13 ms 130.31 ms 1.08 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Order00ManualX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 130.30876666666668 > 127.182759125.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -9.010265626592501 (T) = (0 -128685254.44444446) / Math.Sqrt((806299953434.7192 / (62)) + (1995345845581.4849 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.0611862352079729 = (121265476.4780515 - 128685254.44444446) / 121265476.4780515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedSet - Duration of single invocation 16.03 μs 21.07 μs 1.31 0.14 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.07474632719124 > 16.971043759820628.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 4/22/2022 6:17:37 AM, 4/25/2022 11:05:49 AM, 5/25/2022 1:42:32 AM, 5/26/2022 6:20:15 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -7.167617166513144 (T) = (0 -21832.592917248345) / Math.Sqrt((6076425.424274773 / (62)) + (831529.4561083693 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.25170601917983293 = (17442.26885762994 - 21832.592917248345) / 17442.26885762994 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 79.94 ns 86.13 ns 1.08 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Decimal*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Decimal.TryParse(value: "123456.789")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.1298588249154 > 84.20455378601234.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -43.694114366237486 (T) = (0 -86.12165540907267) / Math.Sqrt((0.775875626670585 / (62)) + (0.004647413973962725 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.06401807266324659 = (80.94003064582297 - 86.12165540907267) / 80.94003064582297 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline b8acc97aba7662204c2a6de8f6090fc115a41c9f
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.TryAddGiventSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 13.83 μs 16.11 μs 1.16 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryAddGiventSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddGiventSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.11326864147089 > 14.771721549823987.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/16/2022 2:55:52 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -73.44911802026502 (T) = (0 -16080.780806182798) / Math.Sqrt((38244.12994990773 / (59)) + (803.8532321309166 / (3))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (59) + (3) - 2, .025) and -0.1604290102708519 = (13857.617022543616 - 16080.780806182798) / 13857.617022543616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 6.71 μs 7.57 μs 1.13 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.MathBenchmarks.Double*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.Min

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.572943730812733 > 6.9240242527170865.
IsChangePoint: Marked as a change because one of 5/1/2022 6:14:18 PM, 5/11/2022 3:41:15 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -31.614393862053273 (T) = (0 -7572.855165901906) / Math.Sqrt((40202.76559630192 / (61)) + (0.09333800977133311 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.12004187391345326 = (6761.2250419193415 - 7572.855165901906) / 6761.2250419193415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Booleans

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBooleans - Duration of single invocation 1.74 ms 1.87 ms 1.08 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Booleans*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Booleans.WriteBooleans(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.8746044444444443 > 1.8288509791666667.
IsChangePoint: Marked as a change because one of 5/20/2022 10:18:01 PM, 5/25/2022 1:42:32 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -65.89241320476378 (T) = (0 -1873574.8765432097) / Math.Sqrt((299264599.2368353 / (62)) + (1567290.4778230279 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.0885437372071666 = (1721175.5600653826 - 1873574.8765432097) / 1721175.5600653826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels May 31, 2022
@tannergooding
Copy link
Member

Possibly related #69878. Noting this has a lot of issues, many may be noise.

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 31, 2022
@tannergooding tannergooding transferred this issue from dotnet/perf-autofiling-issues May 31, 2022
@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 1, 2022
@ghost
Copy link

ghost commented Jun 1, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 14.37 ns 15.55 ns 1.08 0.11 False
TryParseSpan - Duration of single invocation 20.53 ns 22.18 ns 1.08 0.13 False
Parse - Duration of single invocation 20.47 ns 22.81 ns 1.11 0.08 False
TryParseSpan - Duration of single invocation 13.88 ns 15.60 ns 1.12 0.22 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.Parse(value: "12345")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.5548743923072 > 15.104996994035885.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -13.40426032398642 (T) = (0 -15.56223845074927) / Math.Sqrt((0.3997461793558516 / (60)) + (4.075479048722243E-05 / (3))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (3) - 2, .025) and -0.07570467503622394 = (14.467017585681885 - 15.56223845074927) / 14.467017585681885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.1770148518681 > 21.541607506950193.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -10.80960030338378 (T) = (0 -22.56093550737766) / Math.Sqrt((0.2787473236158091 / (62)) + (0.11560187936158357 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11035710442507184 = (20.318630301428488 - 22.56093550737766) / 20.318630301428488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.Parse(value: "2147483647")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.814647795298296 > 21.486482650418488.
IsChangePoint: Marked as a change because one of 4/2/2022 7:01:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -10.171669234588604 (T) = (0 -22.45151356450916) / Math.Sqrt((0.2517145922125223 / (62)) + (0.12021022290261323 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.1051835748492809 = (20.31473691379369 - 22.45151356450916) / 20.31473691379369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryParseSpan(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.596814910317187 > 14.573614871516464.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -14.607255018359659 (T) = (0 -15.744000886858773) / Math.Sqrt((0.4095120601365061 / (62)) + (0.016280183439957862 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11330011734528359 = (14.141740076701943 - 15.744000886858773) / 14.141740076701943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 20.19 ns 24.35 ns 1.21 0.13 False
TryParse - Duration of single invocation 14.69 ns 16.61 ns 1.13 0.11 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.TryParse(value: "4294967295")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.350924724827788 > 20.77327058188295.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -22.410759386694327 (T) = (0 -24.03120853435588) / Math.Sqrt((0.6039468105066214 / (58)) + (0.07850560183156165 / (3))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (58) + (3) - 2, .025) and -0.21708638417156606 = (19.74486679572313 - 24.03120853435588) / 19.74486679572313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.611123351515296 > 15.428355856989672.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -17.621587221432215 (T) = (0 -16.721989773685586) / Math.Sqrt((0.3029141960779122 / (60)) + (0.03013464880684888 / (3))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (60) + (3) - 2, .025) and -0.14871849638918638 = (14.557082371571884 - 16.721989773685586) / 14.557082371571884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetFileName - Duration of single invocation 30.10 ns 33.15 ns 1.10 0.19 False
GetFileNameWithoutExtension - Duration of single invocation 32.28 ns 36.29 ns 1.12 0.15 False
GetDirectoryName - Duration of single invocation 52.35 ns 58.63 ns 1.12 0.13 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetFileName

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.14910856830112 > 31.725914434803297.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -18.711211553238513 (T) = (0 -32.89212759051452) / Math.Sqrt((0.25013250022765565 / (61)) + (0.04966770012052775 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.0890391706491966 = (30.202887533335378 - 32.89212759051452) / 30.202887533335378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetFileNameWithoutExtension

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.289107835551704 > 34.315962475343746.
IsChangePoint: Marked as a change because one of 5/11/2022 3:53:15 AM, 5/17/2022 4:22:41 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.72910225560168 (T) = (0 -36.48132903744981) / Math.Sqrt((0.4577655435142495 / (61)) + (0.0413827433931418 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.10488230228654973 = (33.01829431239132 - 36.48132903744981) / 33.01829431239132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetDirectoryName

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.63079903237569 > 55.01537859752237.
IsChangePoint: Marked as a change because one of 4/22/2022 2:26:33 AM, 4/22/2022 1:40:09 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -33.69073916010231 (T) = (0 -58.60874812444556) / Math.Sqrt((2.085879252477213 / (61)) + (0.0036670744211391624 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.12130489025877728 = (52.268342565526225 - 58.60874812444556) / 52.268342565526225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 4.34 μs 5.31 μs 1.22 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.3097522369464265 > 4.576847852214168.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -9.341961957849813 (T) = (0 -5168.539934723053) / Math.Sqrt((44426.56451919794 / (62)) + (15288.162116411604 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.1598265452989286 = (4456.304225552051 - 5168.539934723053) / 4456.304225552051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 118.15 ns 149.08 ns 1.26 0.01 False
TryParse - Duration of single invocation 113.02 ns 145.46 ns 1.29 0.04 False
Parse - Duration of single invocation 116.84 ns 147.15 ns 1.26 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 149.07886804264805 > 123.65873408194824.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -176.30090851447954 (T) = (0 -149.0854426817671) / Math.Sqrt((1.9879961140862232 / (61)) + (0.001453889677068605 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.2739890133379836 = (117.02254974016435 - 149.0854426817671) / 117.02254974016435 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 145.46473113096087 > 119.01973480938754.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -101.30717916952578 (T) = (0 -145.44026233215502) / Math.Sqrt((5.715957629479416 / (62)) + (0.0029064375805444706 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.27001119658342354 = (114.51888197790502 - 145.44026233215502) / 114.51888197790502 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.Parse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.14664377602486 > 117.53474394091785.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -91.93617956288767 (T) = (0 -146.71752677361312) / Math.Sqrt((5.061785415961881 / (62)) + (0.15736431344164967 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.29779662414423186 = (113.05124704755552 - 146.71752677361312) / 113.05124704755552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline b8acc97aba7662204c2a6de8f6090fc115a41c9f
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterUInt64 - Duration of single invocation 6.33 ns 7.72 ns 1.22 0.01 False
FormatterUInt32 - Duration of single invocation 6.29 ns 7.48 ns 1.19 0.03 False
FormatterUInt32 - Duration of single invocation 13.03 ns 14.26 ns 1.09 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.716702027898865 > 6.645223423740484.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 5:23:46 PM, 5/23/2022 3:43:39 PM, 5/30/2022 10:25:35 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -32.598177930624075 (T) = (0 -7.72627851960429) / Math.Sqrt((0.05569286433136878 / (61)) + (0.00023619372074365139 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.1532265847236033 = (6.699705523573294 - 7.72627851960429) / 6.699705523573294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.478123254655358 > 6.602982892192463.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 3:10:19 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -37.1276609755036 (T) = (0 -7.475300181018473) / Math.Sqrt((0.03365526257398537 / (62)) + (0.0001603703573483267 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.13802117801209834 = (6.568682837762623 - 7.475300181018473) / 6.568682837762623 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.262508361901494 > 13.65358935171701.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/9/2022 5:23:46 PM, 5/23/2022 3:43:39 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -36.325552219793174 (T) = (0 -13.877533769712803) / Math.Sqrt((0.0016081107175497978 / (21)) + (0.02042657686046091 / (43))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (43) - 2, .025) and -0.06550473743110073 = (13.024375474079179 - 13.877533769712803) / 13.024375474079179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 14.25 ns 16.68 ns 1.17 0.16 False
ToString - Duration of single invocation 11.53 ns 13.97 ns 1.21 0.07 False
TryParse - Duration of single invocation 14.90 ns 16.99 ns 1.14 0.14 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.TryParse(value: "32767")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.684172177665495 > 15.264280202338327.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -36.017600746595534 (T) = (0 -16.654300462968283) / Math.Sqrt((0.145794555338259 / (59)) + (0.0008177345806941237 / (3))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (59) + (3) - 2, .025) and -0.12775213454657308 = (14.767695801936437 - 16.654300462968283) / 14.767695801936437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.ToString(value: 32767)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.970511212518261 > 12.113592516296148.
IsChangePoint: Marked as a change because one of 4/15/2022 9:40:04 PM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -54.89665480793696 (T) = (0 -13.96947092043421) / Math.Sqrt((0.1246689760816977 / (61)) + (0.00044405825897137147 / (3))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (3) - 2, .025) and -0.22545577531189687 = (11.399408450197862 - 13.96947092043421) / 11.399408450197862 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int16.TryParse(value: "-32768")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.99456162230395 > 15.689058377667552.
IsChangePoint: Marked as a change because one of 5/16/2022 12:27:40 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -21.917868267261568 (T) = (0 -17.00492699841027) / Math.Sqrt((0.37046122313978247 / (62)) + (8.140462336419778E-05 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11093585876301462 = (15.306848603613009 - 17.00492699841027) / 15.306848603613009 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AppendLine_Strings - Duration of single invocation 224.24 ns 236.50 ns 1.05 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.AppendLine_Strings

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 236.50424263494025 > 236.05565004096457.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -45.86167105878677 (T) = (0 -236.69183426773847) / Math.Sqrt((2.6037737499309066 / (62)) + (0.09574735822032239 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.055606631680901235 = (224.22351959919092 - 236.69183426773847) / 224.22351959919092 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 13.91 μs 15.71 μs 1.13 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<String, String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.712704545454546 > 14.648930340530795.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.26780222645648 (T) = (0 -15670.076459769529) / Math.Sqrt((27561.18341781106 / (62)) + (12418.07817359439 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11176749929286844 = (14094.74235371727 - 15670.076459769529) / 14094.74235371727 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: tannergooding
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.18362, Regression, CoreClr, x64
Milestone: -

@AndyAyersMS AndyAyersMS mentioned this issue Jun 4, 2022
Merged
@AndyAyersMS AndyAyersMS removed the untriaged New issue has not been triaged by the area owner label Jun 4, 2022
@AndyAyersMS AndyAyersMS added this to the 7.0.0 milestone Jun 4, 2022
@AndyAyersMS
Copy link
Member

I have been looking at System.Tests.Perf_UInt32.TryParse(value: "4294967295") and can repro the regression locally with builds just before and at #69878. However the cause of the regression is still not clear to me.

Profiling shows we spend more time in JIT_GetSharedGCThreadStaticBase. Codegen for the benchmark method and get_CurrentInfo are identical. Codegen for TryParseUInt32IntegerStyle is different but profiles as faster.

The static base helper is called from get_CurrentInfo; that and TryParseUInt32IntegerStyle are called from the benchmark method.

image

JIT_GetSharedGCThreadStaticBase is implemented in assembly and (once the class is initialized, which should have happened during the pilot phase) should be quite fast. And the code should be identical in the two versions.

All this suggests some kind of microarchitectural issue, but seems hard to believe it could affect all the benchmarks here and also a similar set on Ubuntu.

Looking at recent history though another possibility suggests itself:
newplot - 2022-06-05T102741 164

The regression is now gone, and the change that removed it was #70144.

That change removed a chk/release diff from the jit, and my investigations above were done with a checked jit, so perhaps I wasn't seeing the actual codegen for some of the methods. (though this is not entirely satisfactory -- as I can reproduce slowdowns even with a checked jit...).

But presumably the problem that was introduced in #68869 could have been exacerbated by #69878 before being (largely) fixed by #70144.

Looking at some of the other reported regressions here we see similar patterns:

newplot - 2022-06-05T103651 768

newplot - 2022-06-05T103844 335

with a few exceptions

newplot - 2022-06-05T104035 711

Looks like I need to dig through all of these and filter the set down to the ones where the regression persists even after #70144.

@AndyAyersMS
Copy link
Member

There were 44 tests flagged as regressions above. Looking at the recent history

Will focus on the regressions that have persisted, in particular these two to start:

newplot - 2022-06-05T111655 627
newplot - 2022-06-05T111809 529

@AndyAyersMS AndyAyersMS mentioned this issue Jun 7, 2022
Closed
@AndyAyersMS
Copy link
Member

Dup of #70159

@ghost ghost locked as resolved and limited conversation to collaborators Jul 11, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@AndyAyersMS AndyAyersMS

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
4 participants
@jeffhandley @jeffschwMSFT @AndyAyersMS @tannergooding