Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 20 Regressions on 1/26/2023 1:49:07 PM #12299

Closed
performanceautofiler bot opened this issue Jan 31, 2023 · 4 comments
Closed

[Perf] Linux/x64: 20 Regressions on 1/26/2023 1:49:07 PM #12299

performanceautofiler bot opened this issue Jan 31, 2023 · 4 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 31, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Fill - Duration of single invocation 15.89 ns 18.23 ns 1.15 0.28 False
Fill - Duration of single invocation 15.61 ns 19.37 ns 1.24 0.26 False
Fill - Duration of single invocation 18.87 ns 21.85 ns 1.16 0.51 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.Fill(Size: 4)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.22942303774202 > 16.94401282348572.
IsChangePoint: Marked as a change because one of 12/12/2022 3:40:04 AM, 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -8.826131128378432 (T) = (0 -18.291722733614854) / Math.Sqrt((0.597089648361213 / (40)) + (0.7389199421638882 / (21))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (21) - 2, .025) and -0.12109846769346165 = (16.315893082298192 - 18.291722733614854) / 16.315893082298192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Fill(Size: 33)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.366387465630357 > 16.479073948393722.
IsChangePoint: Marked as a change because one of 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -12.649269960210454 (T) = (0 -18.700118948610186) / Math.Sqrt((0.32663816906291965 / (41)) + (0.737988322054328 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.16339234246899784 = (16.07378548574941 - 18.700118948610186) / 16.07378548574941 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Fill(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.847610480651728 > 19.977558161644478.
IsChangePoint: Marked as a change because one of 1/25/2023 9:58:39 PM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -3.4018842199944315 (T) = (0 -21.598464087608775) / Math.Sqrt((5.791151625676039 / (38)) + (2.922589658135572 / (24))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (24) - 2, .025) and -0.08988560663650806 = (19.81718444219455 - 21.598464087608775) / 19.81718444219455 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterfaceInterfaceMethodSwitchCallType - Duration of single invocation 4.01 ms 4.41 ms 1.10 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodSwitchCallType

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.414693570833334 > 4.2237307098858174.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -8.496257854521689 (T) = (0 -4272333.6832748735) / Math.Sqrt((8232986133.335594 / (41)) + (9927535642.806257 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.05442009377774101 = (4051832.5745937745 - 4272333.6832748735) / 4051832.5745937745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in Span.IndexerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteViaIndexer2 - Duration of single invocation 8.64 μs 9.51 μs 1.10 0.05 False
KnownSizeArray - Duration of single invocation 7.78 μs 8.32 μs 1.07 0.04 False
ReadOnlyIndexer2 - Duration of single invocation 7.77 μs 8.32 μs 1.07 0.05 False
Indexer2 - Duration of single invocation 7.75 μs 8.32 μs 1.07 0.07 False
Indexer3 - Duration of single invocation 7.76 μs 8.32 μs 1.07 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Span.IndexerBench*'

Payloads

Baseline
Compare

Histogram

Span.IndexerBench.WriteViaIndexer2(length: 1024)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.507399554936653 > 9.0731188284812.
IsChangePoint: Marked as a change because one of 12/4/2022 6:53:17 AM, 12/17/2022 10:44:50 PM, 1/5/2023 9:29:47 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -19.679376655199366 (T) = (0 -9523.631110849281) / Math.Sqrt((57790.029059008324 / (41)) + (7880.536708021728 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.0956470001186694 = (8692.244043763892 - 9523.631110849281) / 8692.244043763892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.KnownSizeArray(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.32000994836277 > 8.147264520972104.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/12/2022 1:34:22 PM, 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -29.86666830024404 (T) = (0 -8319.605545445678) / Math.Sqrt((13322.265365733876 / (41)) + (51.08647367212524 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.06946523245815321 = (7779.220205525675 - 8319.605545445678) / 7779.220205525675 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.ReadOnlyIndexer2(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.315368019920609 > 8.149674045440447.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/12/2022 1:34:22 PM, 12/17/2022 10:44:50 PM, 1/17/2023 11:08:46 AM, 1/17/2023 8:36:05 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -91.22689112205603 (T) = (0 -8319.25586562806) / Math.Sqrt((79.39922361198812 / (41)) + (734.2467453336243 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.07136638437452147 = (7765.089503424131 - 8319.25586562806) / 7765.089503424131 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.Indexer2(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.317424332663604 > 8.144914716965879.
IsChangePoint: Marked as a change because one of 11/18/2022 4:41:14 PM, 12/2/2022 5:21:30 AM, 12/12/2022 1:34:22 PM, 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -14.356450799880351 (T) = (0 -8318.938420178585) / Math.Sqrt((54163.96549844146 / (41)) + (15.074758115142577 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.0669424218442488 = (7796.989087563879 - 8318.938420178585) / 7796.989087563879 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.Indexer3(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.321662033333332 > 8.151384776169643.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/12/2022 1:34:22 PM, 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -77.85425863832748 (T) = (0 -8330.78021366557) / Math.Sqrt((227.60202462078837 / (41)) + (998.6968109554973 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.07308203947911825 = (7763.4140794206105 - 8330.78021366557) / 7763.4140794206105 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexerCheckBoundCheckHoist - Duration of single invocation 387.38 ns 433.93 ns 1.12 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.IndexerCheckBoundCheckHoist

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 433.92630020610494 > 406.749095633249.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/12/2022 1:34:22 PM, 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -13.622908555209593 (T) = (0 -439.5763370145992) / Math.Sqrt((0.8265349241355084 / (41)) + (302.90182236585804 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.1335070983219335 = (387.8020152369198 - 439.5763370145992) / 387.8020152369198 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetByteCount - Duration of single invocation 382.79 μs 409.58 μs 1.07 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishAllAscii)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 409.57729840225556 > 401.0917677172256.
IsChangePoint: Marked as a change because one of 12/2/2022 5:21:30 AM, 1/2/2023 11:09:46 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -19.346805133146066 (T) = (0 -409836.5627438846) / Math.Sqrt((75503760.59324053 / (41)) + (73168.57006926092 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.068514382189128 = (383557.36672839953 - 409836.5627438846) / 383557.36672839953 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualityOperatorBenchmark - Duration of single invocation 22.55 ns 24.02 ns 1.06 0.21 False
GreaterThanOrEqualAllBenchmark - Duration of single invocation 22.41 ns 24.47 ns 1.09 0.28 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Single>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.02035299621684 > 23.81996332173142.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -4.210459842556044 (T) = (0 -24.408612678565802) / Math.Sqrt((0.2649471162153768 / (41)) + (4.276945733892228 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.08586203850420514 = (22.47855787664252 - 24.408612678565802) / 22.47855787664252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.GreaterThanOrEqualAllBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.472185525903463 > 23.590795751455904.
IsChangePoint: Marked as a change because one of 12/19/2022 8:13:55 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -3.4937829913481346 (T) = (0 -24.580621468269772) / Math.Sqrt((2.774797174450088 / (40)) + (2.43356869372179 / (21))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (21) - 2, .025) and -0.06516332689849037 = (23.076856710644428 - 24.580621468269772) / 23.076856710644428 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 2.72 μs 2.88 μs 1.06 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.Array(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8777573787375417 > 2.855152443420681.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/5/2023 9:29:47 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -10.489208246901553 (T) = (0 -2877.2705040226006) / Math.Sqrt((8667.076465060894 / (41)) + (112.04174289607018 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.05671208701551613 = (2722.851890668638 - 2877.2705040226006) / 2722.851890668638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 14.18 μs 15.35 μs 1.08 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.34553849557522 > 14.82593428716927.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -16.179062923325223 (T) = (0 -15469.67666397695) / Math.Sqrt((87806.66066691047 / (41)) + (68054.18243099508 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.08310490347408958 = (14282.713165047564 - 15469.67666397695) / 14282.713165047564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in Devirtualization.EqualityComparer

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ValueTupleCompareWrapped - Duration of single invocation 272.40 ns 291.34 ns 1.07 0.06 False
ValueTupleCompareCached - Duration of single invocation 274.00 ns 290.45 ns 1.06 0.09 False
ValueTupleCompare - Duration of single invocation 274.85 ns 299.69 ns 1.09 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Devirtualization.EqualityComparer*'

Payloads

Baseline
Compare

Histogram

Devirtualization.EqualityComparer.ValueTupleCompareWrapped

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 291.341975828277 > 282.8955595464265.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/3/2022 2:55:34 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -15.794535458441658 (T) = (0 -295.61874292716413) / Math.Sqrt((27.134010924659027 / (41)) + (26.075853009330253 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.07958072114991316 = (273.8273638420353 - 295.61874292716413) / 273.8273638420353 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Devirtualization.EqualityComparer.ValueTupleCompareCached

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.4481937395187 > 283.57510941064896.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 12:08:23 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -17.133674814302157 (T) = (0 -293.5228440526376) / Math.Sqrt((25.60531898535254 / (41)) + (21.333977654948896 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.08080414288944462 = (271.57820034620465 - 293.5228440526376) / 271.57820034620465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Devirtualization.EqualityComparer.ValueTupleCompare

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 299.68595494729806 > 288.970105922138.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/6/2022 2:12:43 AM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -14.200199263920211 (T) = (0 -295.77413742820255) / Math.Sqrt((27.016660548385488 / (41)) + (25.29466983985956 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.07013448174427081 = (276.38968977628315 - 295.77413742820255) / 276.38968977628315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GreaterThanBenchmark - Duration of single invocation 68.89 ns 75.38 ns 1.09 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.GreaterThanBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 75.37816699912675 > 72.36389056963108.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -7.305536353229946 (T) = (0 -76.80449460848779) / Math.Sqrt((8.281273428357675 / (40)) + (14.235707066956385 / (21))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (21) - 2, .025) and -0.0982715632308522 = (69.93215264769975 - 76.80449460848779) / 69.93215264769975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 7bc3c1d31b850723f3d658dea61432d6c8bfab99
Compare 95723eb36afac72806727ff38961fadad71ed057
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArraySetAll - Duration of single invocation 402.03 ns 554.47 ns 1.38 0.21 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 554.4749285545242 > 420.42828381063833.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/26/2023 10:17:57 AM, 1/31/2023 8:24:06 AM falls between 1/22/2023 4:26:49 PM and 1/31/2023 8:24:06 AM.
IsRegressionStdDev: Marked as regression because -41.690466964059425 (T) = (0 -547.3022816366562) / Math.Sqrt((393.7780868219303 / (41)) + (41.313638667540076 / (21))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (21) - 2, .025) and -0.3497531965823374 = (405.4832268761882 - 547.3022816366562) / 405.4832268761882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@lewing
Copy link
Member

lewing commented Jan 31, 2023

looks like dotnet/runtime#81208

@kotlarmilos kotlarmilos mentioned this issue Jan 31, 2023
Closed
@kotlarmilos
Copy link
Member

@BrzVlad it seems that dotnet/runtime#81208 fixed #12052 but created additional regressions.

@BrzVlad
Copy link
Member

BrzVlad commented Mar 29, 2023

Nothing that can be done about that, the change was just small refactoring. Probably just some indirect side effects from compiler

@kotlarmilos
Copy link
Member

Thanks, closing this issue.

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lewing @BrzVlad @kotlarmilos