Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32> #76887

Closed
DrewScoggins opened this issue Oct 11, 2022 · 10 comments
Closed

[Perf] Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32> #76887

DrewScoggins opened this issue Oct 11, 2022 · 10 comments
Assignees
@SingleAccretion
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@DrewScoggins
Copy link
Member

Run Information

Architecture x64
OS ubuntu 18.04
Baseline fb5f07f9c580bdcd5d0726d1391c2a52a01030f8
Compare ae4dc9cef942258098164beae987589c9d569215
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
K_Max_Elements - Duration of single invocation 419.83 ns 442.70 ns 1.05 0.00 False
HeapSort - Duration of single invocation 52.78 μs 55.83 μs 1.06 0.03 False
HeapSort - Duration of single invocation 1.70 μs 1.87 μs 1.10 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.K_Max_Elements(Size: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 442.70104704328384 > 440.7245315214986.
IsChangePoint: Marked as a change because one of 10/2/2022 4:08:26 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -43.82269926441918 (T) = (0 -443.35472183754894) / Math.Sqrt((1.4099182202912361 / (18)) + (7.7759195658518 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.05728774395496787 = (419.3321301343226 - 443.35472183754894) / 419.3321301343226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 55.826369119802784 > 55.34462434146614.
IsChangePoint: Marked as a change because one of 9/3/2022 11:44:47 AM, 9/23/2022 1:16:35 AM, 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -29.270366449344046 (T) = (0 -55822.14208501988) / Math.Sqrt((35281.66684117917 / (14)) + (315056.12359184 / (35))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (35) - 2, .025) and -0.05964015689879864 = (52680.28181226355 - 55822.14208501988) / 52680.28181226355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.866617259837963 > 1.7848844771642938.
IsChangePoint: Marked as a change because one of 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -19.536857311570802 (T) = (0 -1858.4396812229638) / Math.Sqrt((645.542544125046 / (16)) + (408.03389280808665 / (35))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (35) - 2, .025) and -0.08202864547093051 = (1717.5512764859534 - 1858.4396812229638) / 1717.5512764859534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@DrewScoggins DrewScoggins added os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark arch-x64 labels Oct 11, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 11, 2022
@DrewScoggins
Copy link
Member Author

DrewScoggins commented Oct 11, 2022 •
edited
Loading

Seems related to #76155

@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Oct 12, 2022
@ghost
Copy link

ghost commented Oct 12, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline fb5f07f9c580bdcd5d0726d1391c2a52a01030f8
Compare ae4dc9cef942258098164beae987589c9d569215
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
K_Max_Elements - Duration of single invocation 419.83 ns 442.70 ns 1.05 0.00 False
HeapSort - Duration of single invocation 52.78 μs 55.83 μs 1.06 0.03 False
HeapSort - Duration of single invocation 1.70 μs 1.87 μs 1.10 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.K_Max_Elements(Size: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 442.70104704328384 > 440.7245315214986.
IsChangePoint: Marked as a change because one of 10/2/2022 4:08:26 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -43.82269926441918 (T) = (0 -443.35472183754894) / Math.Sqrt((1.4099182202912361 / (18)) + (7.7759195658518 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.05728774395496787 = (419.3321301343226 - 443.35472183754894) / 419.3321301343226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 55.826369119802784 > 55.34462434146614.
IsChangePoint: Marked as a change because one of 9/3/2022 11:44:47 AM, 9/23/2022 1:16:35 AM, 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -29.270366449344046 (T) = (0 -55822.14208501988) / Math.Sqrt((35281.66684117917 / (14)) + (315056.12359184 / (35))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (35) - 2, .025) and -0.05964015689879864 = (52680.28181226355 - 55822.14208501988) / 52680.28181226355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.866617259837963 > 1.7848844771642938.
IsChangePoint: Marked as a change because one of 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -19.536857311570802 (T) = (0 -1858.4396812229638) / Math.Sqrt((645.542544125046 / (16)) + (408.03389280808665 / (35))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (35) - 2, .025) and -0.08202864547093051 = (1717.5512764859534 - 1858.4396812229638) / 1717.5512764859534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: DrewScoggins
Assignees: -
Labels:

os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, area-CodeGen-coreclr, untriaged
Milestone: -

@JulieLeeMSFT
Copy link
Member

Seems related to #76155

@SingleAccretion could you take a look?

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Oct 12, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone Oct 12, 2022
@SingleAccretion
Copy link
Contributor

could you take a look?

Yes, of course.

@JulieLeeMSFT
Copy link
Member

Hi @SingleAccretion, 8/14 is RC1 code snap and 0 bug date. Do you plan to look into these 5 issues assigned to you or do you need help from our team?

@SingleAccretion
Copy link
Contributor

Do you plan to look into these 5 issues assigned to you or do you need help from our team?

I will be doing that, though not sure if I will be get to them all.

@JulieLeeMSFT
Copy link
Member

Do you plan to look into these 5 issues assigned to you or do you need help from our team?

I will be doing that, though not sure if I will be get to them all.

Thanks @SingleAccretion. I talked to @AndyAyersMS and he will help. Please reassign to Andy for the items that you want him to look into by early this week.

@AndyAyersMS
Copy link
Member

The first was "fixed" by dynamic PGO:

newplot - 2023-07-31T115752 166

This next one is just noisy; the June 8 regression is evidently #87115 and the June 29th is likely #88090

newplot - 2023-07-31T115849 746

The last was also "fixed" by dynamic PGO:

newplot - 2023-07-31T120032 761

@DrewScoggins
Copy link
Member Author

DrewScoggins commented Aug 28, 2023 •
edited
Loading

We are still seeing a regression from this change on Windows x86.

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x86_Windows%2010.0.18362%2fSystem.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).HeapSort(Size%3a%20100).html

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x86_Windows%2010.0.18362%2fSystem.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).Dequeue_And_Enqueue(Size%3a%201000).html

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x86_Windows%2010.0.18362%2fSystem.Collections.Tests.Perf_PriorityQueue(Int32%2c%20Int32).Dequeue_And_Enqueue(Size%3a%20100).html

@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 15, 2023
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Sep 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@SingleAccretion SingleAccretion

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
5 participants
@DrewScoggins @jeffschwMSFT @AndyAyersMS @SingleAccretion @JulieLeeMSFT