Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 9 Regressions on 1/18/2023 8:37:25 PM #12058

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Closed

[Perf] Linux/x64: 9 Regressions on 1/18/2023 8:37:25 PM #12058

performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Assignees
@kotlarmilos
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6e773f2b73db1f1b04e5292725a99cf48be84428
Compare 170587e444c43d5c0080f7b2ea454e37463c832a
Diff Diff

Regressions in System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FixedTimeEquals_256Bit_LastBitDifferent - Duration of single invocation 50.50 ns 61.27 ns 1.21 0.03 False
FixedTimeEquals_256Bit_FirstBitDifferent - Duration of single invocation 50.58 ns 60.09 ns 1.19 0.02 False
FixedTimeEquals_256Bit_AllBitsDifferent - Duration of single invocation 53.15 ns 60.25 ns 1.13 0.03 False
FixedTimeEquals_256Bit_Equal - Duration of single invocation 50.53 ns 61.29 ns 1.21 0.03 False
FixedTimeEquals_256Bit_VersusZero - Duration of single invocation 52.00 ns 60.66 ns 1.17 0.04 False
FixedTimeEquals_256Bit_SameReference - Duration of single invocation 51.80 ns 61.51 ns 1.19 0.03 False
FixedTimeEquals_256Bit_CascadingErrors - Duration of single invocation 50.01 ns 61.61 ns 1.23 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals*'

Payloads

Baseline
Compare

Histogram

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_LastBitDifferent

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.27169790321565 > 52.86675009539133.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -121.35798719124521 (T) = (0 -61.35666899388642) / Math.Sqrt((0.07541910316239486 / (29)) + (0.18133043489004924 / (33))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (33) - 2, .025) and -0.21648980536706866 = (50.43747076480629 - 61.35666899388642) / 50.43747076480629 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_FirstBitDifferent

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.09448395023353 > 52.862587304713756.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -67.70100515675146 (T) = (0 -60.763190676300354) / Math.Sqrt((0.3813303903040926 / (29)) + (0.32745861342196103 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.2021615839204418 = (50.544944613969314 - 60.763190676300354) / 50.544944613969314 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_AllBitsDifferent

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.24563973870733 > 53.32856275933576.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -54.91924934006443 (T) = (0 -60.69893810647098) / Math.Sqrt((0.6862014228843244 / (29)) + (0.29572516053455 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.19440039127046807 = (50.819589938267114 - 60.69893810647098) / 50.819589938267114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_Equal

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.29233113725629 > 52.88825314886352.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -68.94336734954771 (T) = (0 -61.08255191859221) / Math.Sqrt((0.19660165859025835 / (29)) + (0.6139823314543417 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.21636964402122158 = (50.2170966028535 - 61.08255191859221) / 50.2170966028535 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_VersusZero

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.6592794382797 > 53.24108168896214.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -95.19843128487825 (T) = (0 -60.55258437755955) / Math.Sqrt((0.2128599780639369 / (29)) + (0.12771194382095818 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.1984794137853586 = (50.52450937501393 - 60.55258437755955) / 50.52450937501393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_SameReference

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.506769979763064 > 53.702211506382646.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -54.28029316122387 (T) = (0 -60.66070366980866) / Math.Sqrt((0.7071081021536701 / (29)) + (0.2568615893194778 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.19035468027831778 = (50.96019251642337 - 60.66070366980866) / 50.96019251642337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_CascadingErrors

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.61144997476021 > 52.64741015065816.
IsChangePoint: Marked as a change because one of 11/4/2022 2:24:58 AM, 12/10/2022 11:25:25 PM, 12/18/2022 12:43:31 PM, 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -85.84923512302822 (T) = (0 -60.623373225473266) / Math.Sqrt((0.16197837915616778 / (29)) + (0.3051683888931382 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.20609844986769202 = (50.264033779434506 - 60.623373225473266) / 50.264033779434506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 5a17537f0baaa70a3b5d25679c4af4dff0182bad
Compare 170587e444c43d5c0080f7b2ea454e37463c832a
Diff Diff

Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 43.43 ns 45.71 ns 1.05 0.01 False
TryFormat - Duration of single invocation 236.66 ns 254.79 ns 1.08 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.TryFormat(address: 143.24.20.36)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.71197809113653 > 45.48600285635275.
IsChangePoint: Marked as a change because one of 11/28/2022 1:52:59 PM, 12/10/2022 11:25:25 PM, 12/20/2022 9:29:18 AM, 1/9/2023 8:29:54 AM, 1/18/2023 6:24:13 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -49.949887883964614 (T) = (0 -45.93287137683134) / Math.Sqrt((0.02587686756740834 / (30)) + (0.06104636891557392 / (33))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (33) - 2, .025) and -0.060036122612863584 = (43.33142087989629 - 45.93287137683134) / 43.33142087989629 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryFormat(address: 1020:3040:5060:7080:9010:1112:1314:1516)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 254.79048071249915 > 246.39098548778605.
IsChangePoint: Marked as a change because one of 12/2/2022 5:47:06 PM, 1/3/2023 6:29:42 AM, 1/9/2023 8:29:54 AM, 1/18/2023 6:24:13 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -16.82409180211483 (T) = (0 -255.15666737628226) / Math.Sqrt((11.051741670489205 / (30)) + (20.637577616413893 / (33))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (33) - 2, .025) and -0.07035518669496829 = (238.38504315950706 - 255.15666737628226) / 238.38504315950706 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@naricc
Copy link

naricc commented Jan 31, 2023

Again someting in here: dotnet/runtime@5a17537...c956f69

I don't understand how any of those caused tihs.

@naricc naricc self-assigned this Jan 31, 2023
@naricc naricc added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Jan 31, 2023
@naricc naricc mentioned this issue Jan 31, 2023
Closed
@kotlarmilos kotlarmilos assigned kotlarmilos and unassigned naricc Feb 28, 2023
@kotlarmilos
Copy link
Member

Looks like transient regression. Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kotlarmilos kotlarmilos

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kotlarmilos @naricc