Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 7 Regressions on 1/18/2023 6:24:13 PM #12057

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Closed

[Perf] Linux/x64: 7 Regressions on 1/18/2023 6:24:13 PM #12057

performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Assignees
@naricc
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6e773f2b73db1f1b04e5292725a99cf48be84428
Compare e6bb67612ab8ad43415c851a1a1dfa6a9057af98
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
PopCount_ulong - Duration of single invocation 745.78 ns 1.10 μs 1.48 0.02 False
Log2_uint - Duration of single invocation 792.43 ns 1.54 μs 1.94 0.02 False
PopCount_uint - Duration of single invocation 667.97 ns 1.10 μs 1.65 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BitOperations.PopCount_ulong

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1031105458530723 > 783.1550193681894.
IsChangePoint: Marked as a change because one of 11/21/2022 6:40:27 PM, 12/14/2022 12:26:48 AM, 12/20/2022 9:29:18 AM, 1/9/2023 10:41:51 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -2111.9709713048283 (T) = (0 -1102.7203318246031) / Math.Sqrt((0.2886627557234696 / (29)) + (0.6318891201679216 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.47830502454670404 = (745.9355907707428 - 1102.7203318246031) / 745.9355907707428 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.Log2_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.538057251441936 > 832.0474005104943.
IsChangePoint: Marked as a change because one of 11/21/2022 6:40:27 PM, 12/14/2022 12:26:48 AM, 12/20/2022 9:29:18 AM, 1/9/2023 10:41:51 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -1575.3541813032227 (T) = (0 -1537.0103018856746) / Math.Sqrt((5.312686496024015 / (29)) + (1.354872173244201 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.9382046009327749 = (793.0072507030359 - 1537.0103018856746) / 793.0072507030359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.PopCount_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.102310320947138 > 701.1053595933454.
IsChangePoint: Marked as a change because one of 11/21/2022 6:40:27 PM, 12/14/2022 12:26:48 AM, 12/20/2022 9:29:18 AM, 1/9/2023 10:41:51 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -2535.5947954461235 (T) = (0 -1102.8049689362138) / Math.Sqrt((0.2780082446687513 / (29)) + (0.6543289202083027 / (33))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (33) - 2, .025) and -0.6510717580677275 = (667.9327918653529 - 1102.8049689362138) / 667.9327918653529 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 6e773f2b73db1f1b04e5292725a99cf48be84428
Compare e6bb67612ab8ad43415c851a1a1dfa6a9057af98
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GreatestCommonDivisor - Duration of single invocation 11.35 ms 13.41 ms 1.18 0.02 False
Add - Duration of single invocation 8.76 μs 9.56 μs 1.09 0.14 False
ToStringD - Duration of single invocation 82.51 ns 144.91 ns 1.76 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.GreatestCommonDivisor(arguments: 65536,65536 bits)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.409527189732144 > 11.827972716622597.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/10/2022 11:25:25 PM, 1/3/2023 6:29:42 AM, 1/9/2023 8:29:54 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -209.64218123980382 (T) = (0 -13416957.860240584) / Math.Sqrt((2734047413.25744 / (29)) + (317035929.07307374 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.18911848920947602 = (11283112.643518103 - 13416957.860240584) / 11283112.643518103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BigInteger.Add(arguments: 65536,65536 bits)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.561363885694842 > 9.262537376054143.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 11/18/2022 7:31:41 AM, 12/8/2022 9:11:55 AM, 1/3/2023 6:29:42 AM, 1/9/2023 8:29:54 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -50.94111998991568 (T) = (0 -9602.248124487107) / Math.Sqrt((4213.842413822254 / (29)) + (4164.773682745184 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.09506947825172485 = (8768.619996438096 - 9602.248124487107) / 8768.619996438096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: 123)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 144.91105920762067 > 86.28743863296096.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -62.86102831854647 (T) = (0 -148.77489169262077) / Math.Sqrt((5.118989588672573 / (9)) + (27.99865811617745 / (53))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (53) - 2, .025) and -0.7938937972096256 = (82.93405770399443 - 148.77489169262077) / 82.93405770399443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6e773f2b73db1f1b04e5292725a99cf48be84428
Compare e6bb67612ab8ad43415c851a1a1dfa6a9057af98
Diff Diff

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 2.47 μs 2.72 μs 1.10 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<String>.List(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7192786293632016 > 2.5933963408767644.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -18.55237299375367 (T) = (0 -2644.100259593009) / Math.Sqrt((619.8886691909236 / (29)) + (1809.235939440255 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.06450677224454067 = (2483.873591539351 - 2644.100259593009) / 2483.873591539351 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@naricc
Copy link

naricc commented Jan 31, 2023

I am confused by this. Here are the changes in the time window, but none of them look like good candidates: dotnet/runtime@5a17537...c956f69

Was it the linker version bump?

Assigning to myself for now.

@naricc naricc self-assigned this Jan 31, 2023
@naricc naricc added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Jan 31, 2023
@naricc naricc mentioned this issue Jan 31, 2023
Closed
@kotlarmilos
Copy link
Member

Mitigated by dotnet/runtime#81695. Noise in some microbenchmarks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@naricc naricc

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kotlarmilos @naricc