Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression from "Mark phase prefetching" #73782

Closed
performanceautofiler bot opened this issue Aug 11, 2022 · 8 comments
Closed

Regression from "Mark phase prefetching" #73782

performanceautofiler bot opened this issue Aug 11, 2022 · 8 comments
Assignees
@PeterSolMS
Labels
arch-arm64 area-GC-coreclr os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline f58a7902a0b4fc9914494de820a7aae8045abd48
Compare b8554beea57afb5d75c55c8ea19ec9965b8917fe
Diff Diff

Regressions in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BenchBitOps - Duration of single invocation 987.31 ms 1.06 secs 1.08 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchBitOps

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0647829177333334 > 989.7809939675001.
IsChangePoint: Marked as a change because one of 8/9/2022 8:37:54 AM, 8/11/2022 10:02:44 AM falls between 8/2/2022 9:32:02 PM and 8/11/2022 10:02:44 AM.
IsRegressionStdDev: Marked as regression because -9.590419168084054 (T) = (0 -1070207212.7276349) / Math.Sqrt((662883780414606.9 / (42)) + (1173360342952634 / (8))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (8) - 2, .025) and -0.12894561950448133 = (947970561.4140848 - 1070207212.7276349) / 947970561.4140848 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline f58a7902a0b4fc9914494de820a7aae8045abd48
Compare b8554beea57afb5d75c55c8ea19ec9965b8917fe
Diff Diff

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 575.53 ns 707.11 ns 1.23 0.28 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 707.1082382724502 > 583.7455950254075.
IsChangePoint: Marked as a change because one of 6/24/2022 3:13:27 AM, 8/9/2022 8:37:54 AM, 8/11/2022 10:02:44 AM falls between 8/2/2022 9:32:02 PM and 8/11/2022 10:02:44 AM.
IsRegressionStdDev: Marked as regression because -10.45194810411 (T) = (0 -684.6831923943578) / Math.Sqrt((881.919006930377 / (42)) + (671.827120731933 / (8))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (8) - 2, .025) and -0.185404420925391 = (577.5946000436348 - 684.6831923943578) / 577.5946000436348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Aug 11, 2022
@kunalspathak kunalspathak changed the title [Perf] Regression on 8/9/2022 11:25:41 AM Regression from "Mark phase prefetching" Aug 11, 2022
@kunalspathak kunalspathak removed refs/heads/main untriaged New issue has not been triaged by the area owner labels Aug 11, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Aug 11, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 11, 2022
@kunalspathak
Copy link
Member

From #73375 @PeterSolMS

@kunalspathak kunalspathak added arch-arm64 os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Aug 11, 2022
@ghost
Copy link

ghost commented Aug 12, 2022

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline f58a7902a0b4fc9914494de820a7aae8045abd48
Compare b8554beea57afb5d75c55c8ea19ec9965b8917fe
Diff Diff

Regressions in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BenchBitOps - Duration of single invocation 987.31 ms 1.06 secs 1.08 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchBitOps

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0647829177333334 > 989.7809939675001.
IsChangePoint: Marked as a change because one of 8/9/2022 8:37:54 AM, 8/11/2022 10:02:44 AM falls between 8/2/2022 9:32:02 PM and 8/11/2022 10:02:44 AM.
IsRegressionStdDev: Marked as regression because -9.590419168084054 (T) = (0 -1070207212.7276349) / Math.Sqrt((662883780414606.9 / (42)) + (1173360342952634 / (8))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (8) - 2, .025) and -0.12894561950448133 = (947970561.4140848 - 1070207212.7276349) / 947970561.4140848 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline f58a7902a0b4fc9914494de820a7aae8045abd48
Compare b8554beea57afb5d75c55c8ea19ec9965b8917fe
Diff Diff

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 575.53 ns 707.11 ns 1.23 0.28 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 707.1082382724502 > 583.7455950254075.
IsChangePoint: Marked as a change because one of 6/24/2022 3:13:27 AM, 8/9/2022 8:37:54 AM, 8/11/2022 10:02:44 AM falls between 8/2/2022 9:32:02 PM and 8/11/2022 10:02:44 AM.
IsRegressionStdDev: Marked as regression because -10.45194810411 (T) = (0 -684.6831923943578) / Math.Sqrt((881.919006930377 / (42)) + (671.827120731933 / (8))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (8) - 2, .025) and -0.185404420925391 = (577.5946000436348 - 684.6831923943578) / 577.5946000436348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

arch-arm64, os-linux, tenet-performance, tenet-performance-benchmarks, area-GC-coreclr, untriaged
Milestone: -

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Aug 15, 2022
@mangod9 mangod9 added this to the 7.0.0 milestone Aug 15, 2022
@mangod9
Copy link
Member

mangod9 commented Aug 18, 2022

Is this actionable as is? The test seems to have had regular spikes in the preceding months?

@kunalspathak
Copy link
Member

The moving average seems to have regressed.

image

@mangod9
Copy link
Member

mangod9 commented Sep 7, 2022

@mrsharm @PeterSolMS can this be closed out, based on your investigation?

@mangod9
Copy link
Member

mangod9 commented Sep 12, 2022

Moving to 8 to continue to investigate why prefetching would cause such regressions.

@mangod9 mangod9 modified the milestones: 7.0.0, 8.0.0 Sep 12, 2022
@mangod9
Copy link
Member

mangod9 commented Aug 2, 2023

Closing these since these have been investigated and improved based on BCN updates.

@mangod9 mangod9 closed this as completed Aug 2, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@PeterSolMS PeterSolMS

Labels
arch-arm64 area-GC-coreclr os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
5 participants
@jeffschwMSFT @kunalspathak @PeterSolMS @dakersnar @mangod9