Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic heap count perf regressions #87014

Closed
performanceautofiler bot opened this issue Jun 1, 2023 · 5 comments
Closed

Dynamic heap count perf regressions #87014

performanceautofiler bot opened this issue Jun 1, 2023 · 5 comments
Labels
arch-arm64 area-GC-coreclr os-windows PGO runtime-coreclr specific to the CoreCLR runtime

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 378.63 ns 451.48 ns 1.19 0.06 False
AllocateArray - Duration of single invocation 176.21 ns 240.06 ns 1.36 0.11 False
AllocateUninitializedArray - Duration of single invocation 152.51 ns 216.20 ns 1.42 0.09 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 451.4849919777149 > 417.33246337973645.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -12.300483702067154 (T) = (0 -448.8153267936087) / Math.Sqrt((157.66566844625675 / (10)) + (4.064186303337012 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.1237107706768952 = (399.40466755805284 - 448.8153267936087) / 399.40466755805284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.06287959149714 > 201.20695919560617.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -17.891866595109473 (T) = (0 -243.96823980558122) / Math.Sqrt((60.11667122191122 / (10)) + (51.97820082886666 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.3163179059028798 = (185.34142756209047 - 243.96823980558122) / 185.34142756209047 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.1952730961082 > 174.84371231759704.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -26.304425455367497 (T) = (0 -217.04011234256407) / Math.Sqrt((43.54454029624648 / (10)) + (1.5513467210148217 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.3458349920432521 = (161.26799617020873 - 217.04011234256407) / 161.26799617020873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateArray - Duration of single invocation 234.50 ns 309.43 ns 1.32 0.12 False
AllocateUninitializedArray - Duration of single invocation 630.54 ns 713.45 ns 1.13 0.02 False
AllocateUninitializedArray - Duration of single invocation 205.68 ns 247.50 ns 1.20 0.08 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 309.42636921378863 > 266.2026636011584.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -22.51211371109258 (T) = (0 -310.4302386863901) / Math.Sqrt((73.19236074573251 / (10)) + (10.161470571407568 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.2629561544335027 = (245.79652872085111 - 310.4302386863901) / 245.79652872085111 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 713.4458116783859 > 682.5010750982198.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -18.597614702447732 (T) = (0 -707.8123762233067) / Math.Sqrt((59.16097295996235 / (10)) + (61.9460605324439 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.09803975613714777 = (644.6145253550358 - 707.8123762233067) / 644.6145253550358 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 247.49783738014705 > 207.88226921271175.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -20.821764934097043 (T) = (0 -249.30922504828592) / Math.Sqrt((47.382854994476055 / (10)) + (19.411808213620578 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.27061343570298096 = (196.21170219277025 - 249.30922504828592) / 196.21170219277025 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToLower_Bytes_Chars - Duration of single invocation 23.48 ns 26.76 ns 1.14 0.15 True
ToUpper_Bytes_Chars - Duration of single invocation 23.06 ns 26.78 ns 1.16 0.25 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 128)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.75858405825367 > 24.481449771884847.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -7.950364658222637 (T) = (0 -26.61790310856316) / Math.Sqrt((1.401941919593238 / (10)) + (0.1075625589421472 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.13079905812608883 = (23.539021294086673 - 26.61790310856316) / 23.539021294086673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 128)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.783557851221513 > 26.131960633946203.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -4.301260013202258 (T) = (0 -26.60903468196683) / Math.Sqrt((3.230228451429849 / (10)) + (0.14287231619255217 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.10338842038906364 = (24.11574581558892 - 26.60903468196683) / 24.11574581558892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_DifferentFirstChar - Duration of single invocation 9.14 ns 10.47 ns 1.14 0.32 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.465237235442851 > 9.614201290192002.
IsChangePoint: Marked as a change because one of 3/10/2023 4:14:40 PM, 3/16/2023 2:30:29 AM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -8.069132154022581 (T) = (0 -11.086277672273589) / Math.Sqrt((0.3541397068576743 / (10)) + (0.17144677872459688 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.19670394158198815 = (9.264010326244964 - 11.086277672273589) / 9.264010326244964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-arm64 os-windows PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jun 1, 2023
@EgorBo EgorBo removed the untriaged New issue has not been triaged by the area owner label Jun 1, 2023
@EgorBo EgorBo changed the title [Perf] Windows/arm64: 9 Regressions on 5/23/2023 8:43:31 PM Dynamic heap count perf regressions Jun 1, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Jun 1, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 1, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 1, 2023
@EgorBo
Copy link
Member

EgorBo commented Jun 1, 2023

We think it's #86245 cc @PeterSolMS

@EgorBo
Copy link
Member

EgorBo commented Jun 1, 2023 •
edited
Loading

Same on Linux-arm64: dotnet/perf-autofiling-issues#18433
Windows-arm64: dotnet/perf-autofiling-issues#18423

@vcsjones vcsjones added area-GC-coreclr and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jun 3, 2023
@ghost
Copy link

ghost commented Jun 3, 2023

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 378.63 ns 451.48 ns 1.19 0.06 False
AllocateArray - Duration of single invocation 176.21 ns 240.06 ns 1.36 0.11 False
AllocateUninitializedArray - Duration of single invocation 152.51 ns 216.20 ns 1.42 0.09 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 451.4849919777149 > 417.33246337973645.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -12.300483702067154 (T) = (0 -448.8153267936087) / Math.Sqrt((157.66566844625675 / (10)) + (4.064186303337012 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.1237107706768952 = (399.40466755805284 - 448.8153267936087) / 399.40466755805284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.06287959149714 > 201.20695919560617.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -17.891866595109473 (T) = (0 -243.96823980558122) / Math.Sqrt((60.11667122191122 / (10)) + (51.97820082886666 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.3163179059028798 = (185.34142756209047 - 243.96823980558122) / 185.34142756209047 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.1952730961082 > 174.84371231759704.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -26.304425455367497 (T) = (0 -217.04011234256407) / Math.Sqrt((43.54454029624648 / (10)) + (1.5513467210148217 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.3458349920432521 = (161.26799617020873 - 217.04011234256407) / 161.26799617020873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateArray - Duration of single invocation 234.50 ns 309.43 ns 1.32 0.12 False
AllocateUninitializedArray - Duration of single invocation 630.54 ns 713.45 ns 1.13 0.02 False
AllocateUninitializedArray - Duration of single invocation 205.68 ns 247.50 ns 1.20 0.08 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 309.42636921378863 > 266.2026636011584.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -22.51211371109258 (T) = (0 -310.4302386863901) / Math.Sqrt((73.19236074573251 / (10)) + (10.161470571407568 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.2629561544335027 = (245.79652872085111 - 310.4302386863901) / 245.79652872085111 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 713.4458116783859 > 682.5010750982198.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -18.597614702447732 (T) = (0 -707.8123762233067) / Math.Sqrt((59.16097295996235 / (10)) + (61.9460605324439 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.09803975613714777 = (644.6145253550358 - 707.8123762233067) / 644.6145253550358 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 1000, pinned: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 247.49783738014705 > 207.88226921271175.
IsChangePoint: Marked as a change because one of 2/28/2023 5:58:27 PM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -20.821764934097043 (T) = (0 -249.30922504828592) / Math.Sqrt((47.382854994476055 / (10)) + (19.411808213620578 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.27061343570298096 = (196.21170219277025 - 249.30922504828592) / 196.21170219277025 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToLower_Bytes_Chars - Duration of single invocation 23.48 ns 26.76 ns 1.14 0.15 True
ToUpper_Bytes_Chars - Duration of single invocation 23.06 ns 26.78 ns 1.16 0.25 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 128)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.75858405825367 > 24.481449771884847.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -7.950364658222637 (T) = (0 -26.61790310856316) / Math.Sqrt((1.401941919593238 / (10)) + (0.1075625589421472 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.13079905812608883 = (23.539021294086673 - 26.61790310856316) / 23.539021294086673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 128)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.783557851221513 > 26.131960633946203.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -4.301260013202258 (T) = (0 -26.60903468196683) / Math.Sqrt((3.230228451429849 / (10)) + (0.14287231619255217 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.10338842038906364 = (24.11574581558892 - 26.60903468196683) / 24.11574581558892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 8163d1643afe2c029c1783b5f92548e6dc6828ff
Compare 94c989db68a702295d4129ec5fe9be9a34419e6c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_DifferentFirstChar - Duration of single invocation 9.14 ns 10.47 ns 1.14 0.32 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.465237235442851 > 9.614201290192002.
IsChangePoint: Marked as a change because one of 3/10/2023 4:14:40 PM, 3/16/2023 2:30:29 AM, 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 5/31/2023 10:16:51 PM falls between 5/22/2023 11:33:11 PM and 5/31/2023 10:16:51 PM.
IsRegressionStdDev: Marked as regression because -8.069132154022581 (T) = (0 -11.086277672273589) / Math.Sqrt((0.3541397068576743 / (10)) + (0.17144677872459688 / (11))) is less than -2.093024054393918 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (11) - 2, .025) and -0.19670394158198815 = (9.264010326244964 - 11.086277672273589) / 9.264010326244964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

arch-arm64, os-windows, area-GC-coreclr, untriaged, runtime-coreclr, PGO
Milestone: -

@mangod9
Copy link
Member

mangod9 commented Jun 3, 2023

@mrsharm as well.

@mrsharm
Copy link
Member

mrsharm commented Jun 12, 2023

@EgorBo, closing this issue as the trends are back to the previous state; fixed by: #87273

image

image

@mrsharm mrsharm closed this as completed Jun 12, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jun 12, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Jul 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-GC-coreclr os-windows PGO runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vcsjones @EgorBo @kunalspathak @mangod9 @mrsharm