Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object> #72194

Closed
performanceautofiler bot opened this issue Jun 30, 2022 · 32 comments
Closed

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object> #72194

performanceautofiler bot opened this issue Jun 30, 2022 · 32 comments
Assignees
@dakersnar
Labels
arch-arm64 area-System.Buffers PGO runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 4.93 ms 5.29 ms 1.07 0.00 False
WriteDeepUtf8 - Duration of single invocation 4.97 ms 5.26 ms 1.06 0.00 False
WriteDeepUtf16 - Duration of single invocation 5.25 ms 5.55 ms 1.06 0.00 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.2924019456521725 > 5.195116339699999.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -62.883214080844944 (T) = (0 -5271577.21509731) / Math.Sqrt((185315176.39245772 / (32)) + (235544487.6445184 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0639908960754851 = (4954532.25637686 - 5271577.21509731) / 4954532.25637686 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.25883959710145 > 5.171207155182354.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -58.01295726028566 (T) = (0 -5242644.245942542) / Math.Sqrt((197904450.00905737 / (32)) + (297749108.7652593 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.06565223672943832 = (4919657.713132181 - 5242644.245942542) / 4919657.713132181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.546248321212121 > 5.525902827765958.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -38.37097812747013 (T) = (0 -5570782.904317904) / Math.Sqrt((269717487.2190087 / (32)) + (695229046.3110473 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.059447521740249476 = (5258196.173008486 - 5570782.904317904) / 5258196.173008486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 2.13 ms 2.41 ms 1.13 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.12 ms 2.39 ms 1.12 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.84 ms 3.09 ms 1.09 0.00 False
WriteBasicUtf16 - Duration of single invocation 3.01 ms 3.28 ms 1.09 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.28 ms 2.55 ms 1.12 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.85 ms 3.12 ms 1.09 0.00 False
WriteBasicUtf8 - Duration of single invocation 3.03 ms 3.27 ms 1.08 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.27 ms 2.56 ms 1.12 0.00 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.413046492905153 > 2.2323625869395554.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -345.39885295672553 (T) = (0 -2415661.9224883527) / Math.Sqrt((11466036.577396711 / (32)) + (4055508.360646513 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.13547246025008064 = (2127450.913213975 - 2415661.9224883527) / 2127450.913213975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3869956682692304 > 2.2289094366944817.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -152.0413470626112 (T) = (0 -2394699.9200306633) / Math.Sqrt((14084111.144689558 / (32)) + (32800381.06979755 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1275283995121029 = (2123848.87251344 - 2394699.9200306633) / 2123848.87251344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0888318115226343 > 2.9820909594602276.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -165.20487949464604 (T) = (0 -3102876.479503347) / Math.Sqrt((9167201.45048848 / (32)) + (26419804.084898308 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09168351763717046 = (2842285.7260125936 - 3102876.479503347) / 2842285.7260125936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.278109338596491 > 3.162077486470761.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -263.1311054031104 (T) = (0 -3276587.661000043) / Math.Sqrt((1732394.4003758235 / (32)) + (11484271.472290196 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08784726663213838 = (3011992.364648777 - 3276587.661000043) / 3011992.364648777 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5517615633284234 > 2.394371170527523.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -401.51257075080696 (T) = (0 -2553628.3245032714) / Math.Sqrt((3139674.47373653 / (32)) + (4375365.40976178 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1197654962107207 = (2280502.77771974 - 2553628.3245032714) / 2280502.77771974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.1181703383333335 > 2.9931869454022983.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -143.7329587356498 (T) = (0 -3112665.3868698566) / Math.Sqrt((5652288.301119157 / (32)) + (37752435.64427367 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09190806637328912 = (2850666.1711991914 - 3112665.3868698566) / 2850666.1711991914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2734030062222224 > 3.180161731956315.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -65.035228030537 (T) = (0 -3281967.6176546924) / Math.Sqrt((5025270.405031393 / (32)) + (178131450.1298167 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08313041922842368 = (3030076.119543044 - 3281967.6176546924) / 3030076.119543044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.555302539027982 > 2.38507844843007.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -351.0938854483454 (T) = (0 -2556497.161926386) / Math.Sqrt((14760943.328055177 / (32)) + (2310805.7294822093 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.12492179784800715 = (2272599.8970035114 - 2556497.161926386) / 2272599.8970035114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MultipleSerial - Duration of single invocation 351.77 ns 378.77 ns 1.08 0.05 False
SingleParallel - Duration of single invocation 7.32 μs 7.97 μs 1.09 0.64 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 378.76595033333336 > 369.10307163333334.
IsChangePoint: Marked as a change because one of 6/19/2022 10:30:09 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -35.54303301900895 (T) = (0 -378.6799848611111) / Math.Sqrt((15.14520057775964 / (32)) + (0.9888827785107522 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.07523194857242699 = (352.18446156094984 - 378.6799848611111) / 352.18446156094984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.971872746499999 > 6.724874629200001.
IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.521013743535269 (T) = (0 -8200.837295708334) / Math.Sqrt((235835.19010466183 / (32)) + (449876.24444322934 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.28216603074986774 = (6396.0806159496515 - 8200.837295708334) / 6396.0806159496515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt32 - Duration of single invocation 9.82 ns 15.97 ns 1.63 0.16 False
FormatterInt64 - Duration of single invocation 10.42 ns 16.10 ns 1.55 0.20 False
FormatterInt32 - Duration of single invocation 3.01 ns 6.25 ns 2.08 0.45 False
FormatterInt32 - Duration of single invocation 23.21 ns 28.53 ns 1.23 0.02 False
FormatterInt64 - Duration of single invocation 48.79 ns 54.49 ns 1.12 0.00 False
FormatterInt32 - Duration of single invocation 21.64 ns 27.80 ns 1.28 0.02 False
FormatterInt64 - Duration of single invocation 51.93 ns 55.85 ns 1.08 0.00 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.972661986380423 > 10.892674528760839.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -54.720373930164925 (T) = (0 -15.819675463730695) / Math.Sqrt((0.23147261689839174 / (32)) + (0.041780717070086536 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5577714757301261 = (10.155324904967866 - 15.819675463730695) / 10.155324904967866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.09793687781935 > 10.824888950439226.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -85.17981573969624 (T) = (0 -16.084501663821538) / Math.Sqrt((0.09366564672038877 / (32)) + (0.019441132352970018 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5554691210964263 = (10.340611360053114 - 16.084501663821538) / 10.340611360053114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.250267075893282 > 3.352759540047291.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -40.262601231643046 (T) = (0 -6.38887748599731) / Math.Sqrt((0.05214754665397373 / (32)) + (0.05028820581828033 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.9259796783391032 = (3.3172091885760975 - 6.38887748599731) / 3.3172091885760975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.534069930021154 > 24.45641570850646.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -74.28006930773032 (T) = (0 -28.327018439232784) / Math.Sqrt((0.0551718473136399 / (32)) + (0.03327214186978605 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.21335989886260198 = (23.345932617178466 - 28.327018439232784) / 23.345932617178466 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.49095898922284 > 51.227091925254726.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -42.96161504376527 (T) = (0 -54.068247923961614) / Math.Sqrt((0.13240447345404585 / (32)) + (0.14339452112148282 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.11207604377403953 = (48.61920030258977 - 54.068247923961614) / 48.61920030258977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.795311175006713 > 22.797346430260795.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -110.16261371835587 (T) = (0 -27.615813537490684) / Math.Sqrt((0.02893923834467118 / (32)) + (0.024010447309944224 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.27390750635811145 = (21.67803659187132 - 27.615813537490684) / 21.67803659187132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.848231463165604 > 54.11531003142006.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -30.30745666185831 (T) = (0 -56.216933586070546) / Math.Sqrt((0.16993580402529118 / (32)) + (0.24770725649393255 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0951113723664043 = (51.33444415301112 - 56.216933586070546) / 51.33444415301112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Collections.Concurrent.Count<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 7.78 ns 12.44 ns 1.60 0.05 False
Queue_EnqueueCountDequeue - Duration of single invocation 33.47 ns 42.69 ns 1.28 0.11 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.Count&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.Count<String>.Queue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.440158083173902 > 8.179443964478883.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -146.98536372924585 (T) = (0 -12.371144724936286) / Math.Sqrt((0.014615805850382501 / (32)) + (0.0061045493574273715 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5852219552120929 = (7.8040457894624 - 12.371144724936286) / 7.8040457894624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Concurrent.Count&lt;String&gt;.Queue_EnqueueCountDequeue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.689222531441224 > 35.04501625853797.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -76.91165895557451 (T) = (0 -42.74227963262908) / Math.Sqrt((0.4322036508108061 / (32)) + (0.005601279562968299 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.2701748915241361 = (33.650704259584934 - 42.74227963262908) / 33.650704259584934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FirstSpanTenSegments - Duration of single invocation 6.66 ns 8.73 ns 1.31 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Char>.FirstSpanTenSegments

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.733886982660364 > 6.9937097877618255.
IsChangePoint: Marked as a change because one of 5/20/2022 10:18:01 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -179.81577296993908 (T) = (0 -8.75725016440443) / Math.Sqrt((0.0021192516157491448 / (32)) + (0.0008248153498479827 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.3132697641219422 = (6.668279742402785 - 8.75725016440443) / 6.668279742402785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FirstSpanTenSegments - Duration of single invocation 6.00 ns 7.47 ns 1.25 0.04 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Byte>.FirstSpanTenSegments

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.4695475393018675 > 6.297132596697012.
IsChangePoint: Marked as a change because one of 5/11/2022 7:18:06 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -77.5875013494363 (T) = (0 -7.3390959024405085) / Math.Sqrt((0.0007146562241878547 / (32)) + (0.0033294826669831956 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.22405752032249257 = (5.995711623508457 - 7.3390959024405085) / 5.995711623508457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added arm64 PGO untriaged New issue has not been triaged by the area owner labels Jun 30, 2022
@dakersnar dakersnar transferred this issue from dotnet/perf-autofiling-issues Jul 14, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dakersnar dakersnar changed the title [Perf] Changes at 6/27/2022 5:19:23 PM Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object> Jul 14, 2022
@dakersnar
Copy link
Contributor

dakersnar commented Jul 14, 2022 •
edited
Loading

All the above regressions have been solved except for System.Buffers.Tests.RentReturnArrayPoolTests MultipleSerial and SingleParallel

Edit: I was mistaken, all benchmarks listed above have regressed

@dakersnar dakersnar added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jul 14, 2022
@ghost
Copy link

ghost commented Jul 15, 2022

Tagging subscribers to this area: @dotnet/area-system-buffers
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 4.93 ms 5.29 ms 1.07 0.00 False
WriteDeepUtf8 - Duration of single invocation 4.97 ms 5.26 ms 1.06 0.00 False
WriteDeepUtf16 - Duration of single invocation 5.25 ms 5.55 ms 1.06 0.00 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.2924019456521725 > 5.195116339699999.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -62.883214080844944 (T) = (0 -5271577.21509731) / Math.Sqrt((185315176.39245772 / (32)) + (235544487.6445184 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0639908960754851 = (4954532.25637686 - 5271577.21509731) / 4954532.25637686 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.25883959710145 > 5.171207155182354.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -58.01295726028566 (T) = (0 -5242644.245942542) / Math.Sqrt((197904450.00905737 / (32)) + (297749108.7652593 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.06565223672943832 = (4919657.713132181 - 5242644.245942542) / 4919657.713132181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.546248321212121 > 5.525902827765958.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -38.37097812747013 (T) = (0 -5570782.904317904) / Math.Sqrt((269717487.2190087 / (32)) + (695229046.3110473 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.059447521740249476 = (5258196.173008486 - 5570782.904317904) / 5258196.173008486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 2.13 ms 2.41 ms 1.13 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.12 ms 2.39 ms 1.12 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.84 ms 3.09 ms 1.09 0.00 False
WriteBasicUtf16 - Duration of single invocation 3.01 ms 3.28 ms 1.09 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.28 ms 2.55 ms 1.12 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.85 ms 3.12 ms 1.09 0.00 False
WriteBasicUtf8 - Duration of single invocation 3.03 ms 3.27 ms 1.08 0.00 False
WriteBasicUtf8 - Duration of single invocation 2.27 ms 2.56 ms 1.12 0.00 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.413046492905153 > 2.2323625869395554.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -345.39885295672553 (T) = (0 -2415661.9224883527) / Math.Sqrt((11466036.577396711 / (32)) + (4055508.360646513 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.13547246025008064 = (2127450.913213975 - 2415661.9224883527) / 2127450.913213975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3869956682692304 > 2.2289094366944817.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -152.0413470626112 (T) = (0 -2394699.9200306633) / Math.Sqrt((14084111.144689558 / (32)) + (32800381.06979755 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1275283995121029 = (2123848.87251344 - 2394699.9200306633) / 2123848.87251344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0888318115226343 > 2.9820909594602276.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -165.20487949464604 (T) = (0 -3102876.479503347) / Math.Sqrt((9167201.45048848 / (32)) + (26419804.084898308 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09168351763717046 = (2842285.7260125936 - 3102876.479503347) / 2842285.7260125936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.278109338596491 > 3.162077486470761.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -263.1311054031104 (T) = (0 -3276587.661000043) / Math.Sqrt((1732394.4003758235 / (32)) + (11484271.472290196 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08784726663213838 = (3011992.364648777 - 3276587.661000043) / 3011992.364648777 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5517615633284234 > 2.394371170527523.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -401.51257075080696 (T) = (0 -2553628.3245032714) / Math.Sqrt((3139674.47373653 / (32)) + (4375365.40976178 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1197654962107207 = (2280502.77771974 - 2553628.3245032714) / 2280502.77771974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.1181703383333335 > 2.9931869454022983.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -143.7329587356498 (T) = (0 -3112665.3868698566) / Math.Sqrt((5652288.301119157 / (32)) + (37752435.64427367 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09190806637328912 = (2850666.1711991914 - 3112665.3868698566) / 2850666.1711991914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2734030062222224 > 3.180161731956315.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -65.035228030537 (T) = (0 -3281967.6176546924) / Math.Sqrt((5025270.405031393 / (32)) + (178131450.1298167 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08313041922842368 = (3030076.119543044 - 3281967.6176546924) / 3030076.119543044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.555302539027982 > 2.38507844843007.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -351.0938854483454 (T) = (0 -2556497.161926386) / Math.Sqrt((14760943.328055177 / (32)) + (2310805.7294822093 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.12492179784800715 = (2272599.8970035114 - 2556497.161926386) / 2272599.8970035114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MultipleSerial - Duration of single invocation 351.77 ns 378.77 ns 1.08 0.05 False
SingleParallel - Duration of single invocation 7.32 μs 7.97 μs 1.09 0.64 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 378.76595033333336 > 369.10307163333334.
IsChangePoint: Marked as a change because one of 6/19/2022 10:30:09 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -35.54303301900895 (T) = (0 -378.6799848611111) / Math.Sqrt((15.14520057775964 / (32)) + (0.9888827785107522 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.07523194857242699 = (352.18446156094984 - 378.6799848611111) / 352.18446156094984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.971872746499999 > 6.724874629200001.
IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.521013743535269 (T) = (0 -8200.837295708334) / Math.Sqrt((235835.19010466183 / (32)) + (449876.24444322934 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.28216603074986774 = (6396.0806159496515 - 8200.837295708334) / 6396.0806159496515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt32 - Duration of single invocation 9.82 ns 15.97 ns 1.63 0.16 False
FormatterInt64 - Duration of single invocation 10.42 ns 16.10 ns 1.55 0.20 False
FormatterInt32 - Duration of single invocation 3.01 ns 6.25 ns 2.08 0.45 False
FormatterInt32 - Duration of single invocation 23.21 ns 28.53 ns 1.23 0.02 False
FormatterInt64 - Duration of single invocation 48.79 ns 54.49 ns 1.12 0.00 False
FormatterInt32 - Duration of single invocation 21.64 ns 27.80 ns 1.28 0.02 False
FormatterInt64 - Duration of single invocation 51.93 ns 55.85 ns 1.08 0.00 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.972661986380423 > 10.892674528760839.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -54.720373930164925 (T) = (0 -15.819675463730695) / Math.Sqrt((0.23147261689839174 / (32)) + (0.041780717070086536 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5577714757301261 = (10.155324904967866 - 15.819675463730695) / 10.155324904967866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.09793687781935 > 10.824888950439226.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -85.17981573969624 (T) = (0 -16.084501663821538) / Math.Sqrt((0.09366564672038877 / (32)) + (0.019441132352970018 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5554691210964263 = (10.340611360053114 - 16.084501663821538) / 10.340611360053114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.250267075893282 > 3.352759540047291.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -40.262601231643046 (T) = (0 -6.38887748599731) / Math.Sqrt((0.05214754665397373 / (32)) + (0.05028820581828033 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.9259796783391032 = (3.3172091885760975 - 6.38887748599731) / 3.3172091885760975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.534069930021154 > 24.45641570850646.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -74.28006930773032 (T) = (0 -28.327018439232784) / Math.Sqrt((0.0551718473136399 / (32)) + (0.03327214186978605 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.21335989886260198 = (23.345932617178466 - 28.327018439232784) / 23.345932617178466 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.49095898922284 > 51.227091925254726.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -42.96161504376527 (T) = (0 -54.068247923961614) / Math.Sqrt((0.13240447345404585 / (32)) + (0.14339452112148282 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.11207604377403953 = (48.61920030258977 - 54.068247923961614) / 48.61920030258977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.795311175006713 > 22.797346430260795.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -110.16261371835587 (T) = (0 -27.615813537490684) / Math.Sqrt((0.02893923834467118 / (32)) + (0.024010447309944224 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.27390750635811145 = (21.67803659187132 - 27.615813537490684) / 21.67803659187132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.848231463165604 > 54.11531003142006.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -30.30745666185831 (T) = (0 -56.216933586070546) / Math.Sqrt((0.16993580402529118 / (32)) + (0.24770725649393255 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0951113723664043 = (51.33444415301112 - 56.216933586070546) / 51.33444415301112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Collections.Concurrent.Count<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 7.78 ns 12.44 ns 1.60 0.05 False
Queue_EnqueueCountDequeue - Duration of single invocation 33.47 ns 42.69 ns 1.28 0.11 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.Count&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.Count<String>.Queue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.440158083173902 > 8.179443964478883.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -146.98536372924585 (T) = (0 -12.371144724936286) / Math.Sqrt((0.014615805850382501 / (32)) + (0.0061045493574273715 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5852219552120929 = (7.8040457894624 - 12.371144724936286) / 7.8040457894624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Concurrent.Count&lt;String&gt;.Queue_EnqueueCountDequeue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.689222531441224 > 35.04501625853797.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -76.91165895557451 (T) = (0 -42.74227963262908) / Math.Sqrt((0.4322036508108061 / (32)) + (0.005601279562968299 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.2701748915241361 = (33.650704259584934 - 42.74227963262908) / 33.650704259584934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline c9f80fdde29c71ac87971383c77a1a5e665fadb9
Compare a38b50c3a2f828458d15e68c78460db3ec7e5cda
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FirstSpanTenSegments - Duration of single invocation 6.66 ns 8.73 ns 1.31 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Char>.FirstSpanTenSegments

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.733886982660364 > 6.9937097877618255.
IsChangePoint: Marked as a change because one of 5/20/2022 10:18:01 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -179.81577296993908 (T) = (0 -8.75725016440443) / Math.Sqrt((0.0021192516157491448 / (32)) + (0.0008248153498479827 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.3132697641219422 = (6.668279742402785 - 8.75725016440443) / 6.668279742402785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: dakersnar
Labels:

area-System.Buffers, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, PGO, Regression, CoreClr, arm64, ubuntu 20.04
Milestone: -

@jeffhandley jeffhandley added this to the 7.0.0 milestone Aug 2, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Aug 2, 2022
@mrsharm
Copy link
Member

mrsharm commented Aug 12, 2022

This regression was detected by our August Performance report:

image

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

Result Ratio Alloc Delta Operating System Bit Processor Name
Faster 1.26 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.68 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 1.54 +0 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 1.03 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.79 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 1.02 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 1.21 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Same 0.93 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Slower 0.80 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 0.90 +0 Windows 11 X64 AMD Ryzen 9 3950X
Slower 0.87 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.81 +0 Windows 11 X64 AMD Ryzen 9 5950X
Slower 0.84 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.96 +0 Windows 11 X64 Intel Core i9-10900K CPU 3.70GHz
Slower 0.66 +0 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Same 1.01 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 0.92 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Same 0.89 +0 ubuntu 18.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 1.00 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.88 +0 ubuntu 20.04 X64 Intel Core i9-10900K CPU 3.70GHz
Same 0.06 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.08 +0 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 7.30 +0 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 1.03 +0 macOS Big Sur 11.6.8 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Same 0.93 +0 macOS Monterey 12.3.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 0.96 +0 macOS Monterey 12.4 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

@dakersnar
Copy link
Contributor

@AndyAyersMS Thoughts on these regressions? Given this is a PGO update should we assume this is by design and close this issue? Or should we investigate this further.

Specifically, System.Buffers.Tests.RentReturnArrayPoolTests MultipleSerial and SingleParallel and the one Mukund flagged above.

@mrsharm mrsharm mentioned this issue Aug 12, 2022
Closed
24 tasks
@AndyAyersMS
Copy link
Member

Have we isolated this to the PGO udpate? If not, we should probably drill in.

@dakersnar
Copy link
Contributor

@AndyAyersMS I'm unsure if we have 100% isolated this to PGO. Looking at the commit range (52f4e7c...8fe39db), I've gotten some more insight.

First of all, considering these tests have not regressed on Windows x64, it seems like this is arm64 specific. In my opinion, that rules out commits like #71302 which should affect all configs, right? On that note, could the PGO update only affect arm64?

Otherwise, the only other arm64 specific commit I see in that range is #71306, which doesn't seem like it should affect this.

@dakersnar
Copy link
Contributor

Actually, looking at your comment here dotnet/perf-autofiling-issues#6480 (comment), it could be caused by #71161.

@AndyAyersMS
Copy link
Member

Seems possible but unlikely -- diffs from #71161 are no longer available so hard to say for sure.

@jakobbotsch any chance you could take a look?

In the lab we only see this regression (June 27) on the ampere ubuntu arm64 runs, so it's not all arm64 HW either.

ubuntu ampere

newplot - 2022-08-15T113714 493

windows ampere

newplot - 2022-08-15T113823 346

windows surface pro x
newplot - 2022-08-15T113635 920

ubuntu centriq
newplot - 2022-08-15T113929 615

@jakobbotsch
Copy link
Member

jakobbotsch commented Aug 16, 2022 •
edited
Loading

Hottest methods via PGO data (sum of basic block counts):

System.Buffers.ConfigurableArrayPool`1+Bucket[System.__Canon].Return: 152000000
System.Buffers.ConfigurableArrayPool`1+Bucket[System.__Canon].Rent: 136800008
System.Threading.SpinLock.Enter: 121600000
System.Buffers.ConfigurableArrayPool`1[System.__Canon].Rent: 106400000
System.Buffers.Tests.RentReturnArrayPoolTests`1+<MultipleSerial>d__23[System.__Canon].MoveNext: 104500133
System.Numerics.BitOperations.Log2: 91200123
System.Buffers.ConfigurableArrayPool`1[System.__Canon].Return: 91200000
System.Threading.SpinLock.Exit: 60800000
System.ArgumentNullException.ThrowIfNull: 30413678
System.Type.op_Equality: 153018
System.Runtime.CompilerServices.CastHelpers.IsInstanceOfClass: 113322
System.Diagnostics.Debug.Assert: 82844
Perfolizer.Mathematics.Distributions.StudentDistribution.StudentTwoTail: 69354
...

Comparing 52f4e7c (left) to 8fe39db (right): https://www.diffchecker.com/BfH1vTmK
Definitely seems PGO related:

 ; Assembly listing for method Bucket[__Canon][System.__Canon]:Return(System.__Canon[]):this
 ; Emitting BLENDED_CODE for generic ARM64 CPU - Unix
 ; Tier-1 compilation
 ; optimized code
 ; fp based frame
 ; fully interruptible
 ; No PGO data
-; 2 inlinees with PGO data; 6 single block inlinees; 0 inlinees without PGO data
+; 1 inlinees with PGO data; 6 single block inlinees; 1 inlinees without PGO data
 ; Assembly listing for method Bucket[__Canon][System.__Canon]:Rent():System.__Canon[]:this
 ; Emitting BLENDED_CODE for generic ARM64 CPU - Unix
 ; Tier-1 compilation
 ; optimized code
 ; fp based frame
 ; fully interruptible
 ; No PGO data
-; 2 inlinees with PGO data; 5 single block inlinees; 0 inlinees without PGO data
+; 1 inlinees with PGO data; 5 single block inlinees; 1 inlinees without PGO data
 ; Final local variable assignments

and these two methods are the only ones with significant diffs (looks like some BB reordering).

@AndyAyersMS
Copy link
Member

Interesting. We should try and root cause what's behind this...

  • PGO collections differed?
  • Runtime failed to match up PGO data?
  • Jit failed to use PGO data?

I'll start in on this, but @jakobbotsch feel free to jump in if you have some of this lying around.

@jakobbotsch
Copy link
Member

I'll start in on this, but @jakobbotsch feel free to jump in if you have some of this lying around.

I'm currently deep in some of my quality week work, but I can try to look into the dumps tomorrow if you haven't root caused it by then.

One thing I just realized is that I did the above investigation with a checked SPC, so it may have contributed to failures in matching up PGO data... (though I would expect to see it in both base and diff)

@AndyAyersMS
Copy link
Member

Ok, I'll let you know how far I get.

@AndyAyersMS
Copy link
Member

My local profile shows:


03.95%   1.71E+06    ?        Unknown
35.74%   1.547E+07   Tier-1   [System.Private.CoreLib]EventSource.WriteEventWithRelatedActivityIdCore(int32,value class System.Guid*,int32,value class EventData*)
18.67%   8.08E+06    Tier-1   [System.Private.CoreLib]System.Buffers.ConfigurableArrayPool`1+Bucket[System.__Canon].Return(!0[])
16.01%   6.93E+06    Tier-1   [System.Private.CoreLib]System.Buffers.ConfigurableArrayPool`1+Bucket[System.__Canon].Rent()
05.85%   2.53E+06    Tier-1   [MicroBenchmarks]System.Buffers.Tests.RentReturnArrayPoolTests`1+<MultipleSerial>d__23[System.__Canon].MoveNext()
04.30%   1.86E+06    Tier-1   [System.Private.CoreLib]System.Buffers.ConfigurableArrayPool`1[System.__Canon].Return(!0[],bool)
03.95%   1.71E+06    OSR      [MicroBenchmarks]System.Buffers.Tests.RentReturnArrayPoolTests`1+<MultipleSerial>d__23[System.__Canon].MoveNext()
03.74%   1.62E+06    Tier-1   [System.Private.CoreLib]System.Buffers.ConfigurableArrayPool`1[System.__Canon].Rent(int32)
02.70%   1.17E+06    native   coreclr.dll

So indeed, bucket Rent/Return are dominant.

A few interesting tidbits:

  • The EventSource method is likely is an artifact of -p ETW -- seems like this provider should not be enabled?
  • The OSR version of MoveNext is there for a while during the actual intervals. Not sure it matters.
  • The autogenerated runner is getting tiered up (not seen in the above), which is undesirable. I thought I had changed all those methods to be marked with AggressiveOptimization (Fix optimization of action methods for coreclr BenchmarkDotNet#1935) but evidently, I missed some cases. This can cause increased noise in BDN data as the overhead measurement may be misleading.

@AndyAyersMS
Copy link
Member

With release builds (and checked jit) I don't see any codegen diff at Tier1 for Bucket.Rent or Bucket.Return. There are some small PGO count diffs for some inlined spinlock methods, but they wash out.

Going to check some of the other methods noted above.

@AndyAyersMS
Copy link
Member

Also very odd:

  • a run with a checked jit (but everything else release) seems to run much faster than with a release jit.
  • dumping ConfigurableArrayPool.Rent or .Return (with checked jit/release runtime) causes the jit to crash

@AndyAyersMS
Copy link
Member

AndyAyersMS commented Aug 17, 2022 •
edited
Loading

One small asm diff in ConfigurableArrayPool1:Rent(int):ref:this` -- LSRA perhaps. But otherwise all the rent/returns line up.
rr.base.dasm.txt
rr.diff.dasm.txt

as do the move nexts

mn.base.dasm.txt
mn.diff.dasm.txt

@jakobbotsch
Copy link
Member

jakobbotsch commented Aug 17, 2022 •
edited
Loading

I still see significant diffs in Bucket:Rent and Bucket:Return with a release build.

Method Job Toolchain RentalSize ManipulateArray Async UseSharedPool Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
MultipleSerial Job-OMYRHA /linux-arm64-52f4e7c567-release/corerun 4096 False False False 1.546 us 0.0303 us 0.0324 us 1.522 us 1.518 us 1.593 us 1.00 0.00 - NA
MultipleSerial Job-QUPOOX /linux-arm64-8fe39db6203-release/corerun 4096 False False False 1.726 us 0.0053 us 0.0049 us 1.726 us 1.712 us 1.732 us 1.11 0.02 - NA

Diffs: https://www.diffchecker.com/D7VI2RXp (52f4e7c on the left, 8fe39db on the right)
Note that incremental builds with new PGO data does not work in the backwards direction, so you may need a clean build.

@jakobbotsch
Copy link
Member

There's no PGO info for SpinLock.Enter in the new PGO data according to the dump:

 *************** Inline @[000018] Starting PHASE Profile incorporation
-Have static profile data: 5 schema records (schema at 00000000D1FFAB1E, data at 00000000D1FFAB1E)
-Profile summary: 2 runs, 0 block probes, 4 edge probes, 0 class profiles, 0 method profiles, 0 other records
-
-Reconstructing block counts from sparse edge instrumentation
-... adding known edge BB15 -> BB17: weight 0
-... adding known edge BB16 -> BB17: weight 0
-... adding known edge BB16 -> BB18: weight 2
-... adding known edge BB18 -> BB14: weight 2
-
-New BlockSet epoch 2, # of blocks (including unused BB00): 19, bitset array size: 1 (short)
- ... unknown edge BB14 -> BB15
- ... unknown edge BB14 -> BB17
- ... unknown edge BB17 -> BB18
- ... unknown edge BB15 -> BB16
-
-Solver: 5 blocks, 5 unknown; 8 edges, 4 unknown, 0 zero (and so ignored)
-
-Pass [1]: 5 unknown blocks, 4 unknown edges
-BB18: 1 incoming unknown, 0 outgoing unknown
-BB18: all outgoing edge weights known, summming...
-  BB18 -> BB14 has weight 2
-BB18: all outgoing edge weights known, sum is 2
-BB17 -> BB18: target block weight and all other incoming edge weights known, so weight is 0
-BB17: 1 incoming unknown, 0 outgoing unknown
-BB17: all outgoing edge weights known, summming...
-  BB17 -> BB18 has weight 0
-BB17: all outgoing edge weights known, sum is 0
-BB14 -> BB17: target block weight and all other incoming edge weights known, so weight is 0
-BB16: 1 incoming unknown, 0 outgoing unknown
-BB16: all outgoing edge weights known, summming...
-  BB16 -> BB17 has weight 0
-  BB16 -> BB18 has weight 2
-BB16: all outgoing edge weights known, sum is 2
-BB15 -> BB16: target block weight and all other incoming edge weights known, so weight is 2
-BB15: 1 incoming unknown, 0 outgoing unknown
-BB15: all outgoing edge weights known, summming...
-  BB15 -> BB17 has weight 0
-  BB15 -> BB16 has weight 2
-BB15: all outgoing edge weights known, sum is 2
-BB14 -> BB15: target block weight and all other incoming edge weights known, so weight is 2
-BB14: 0 incoming unknown, 0 outgoing unknown
-BB14: all incoming edge weights known, summming...
-  BB18 -> BB14 has weight 2
-BB14: all incoming edge weights known, sum is 2
-
-Solver: converged in 1 passes
+BBOPT set, but no profile data available (hr=80004001)
 Computing inlinee profile scale:
+   ... no callee profile data, will use non-pgo weight to scale
    ... call site not profiled, will use non-pgo weight to scale
-   call site count 100 callee entry count 2 scale 50
+   call site count 100 callee entry count 100 scale 1
 Scaling inlinee blocks
 
 *************** Inline @[000018] Finishing PHASE Profile incorporation

In the base we use the PGO data to reverse a branch:

@@ -5,10 +5,10 @@ Initial BasicBlocks
 BBnum BBid ref try hnd preds           weight   IBC  lp [IL range]     [jump]      [EH region]         [flags]
 -----------------------------------------------------------------------------------------------------------------------------------------
 BB01 [0000]  1                             1           [000..00D)                                     i 
-BB02 [0001]  1  0    BB01                  1    100    [00D..025)-> BB05 ( cond ) T0      try {       keep i try idxlen nullcheck IBC 
-BB03 [0014]  1  0    BB02                  0.50  50    [00D..00E)-> BB05 ( cond ) T0                  i IBC 
-BB04 [0015]  1  0    BB03                  0.50  50    [00D..00E)-> BB06 ( cond ) T0                  i IBC 
-BB05 [0016]  3  0    BB02,BB03,BB04        0      0    [00D..00E)                 T0                  i rare hascall gcsafe IBC 
+BB02 [0001]  1  0    BB01                  1           [00D..025)-> BB05 ( cond ) T0      try {       keep i try idxlen nullcheck 
+BB03 [0014]  1  0    BB02                  0.50        [00D..00E)-> BB05 ( cond ) T0                  i 
+BB04 [0015]  1  0    BB03                  0.50        [00D..00E)-> BB06 ( cond ) T0                  i 
+BB05 [0016]  3  0    BB02,BB03,BB04        0.50        [00D..00E)                 T0                  i hascall gcsafe 
 BB06 [0018]  2  0    BB04,BB05             1           [???..???)-> BB08 ( cond ) T0                  keep internal idxlen 
 BB07 [0002]  1  0    BB06                  0.50        [025..04C)                 T0      }           i idxlen 
 BB08 [0028]  2       BB06,BB07             1           [04C..04F)-> BB12 ( cond )                     keep i cfb 
@@ -30,50 +30,6 @@ BB22 [0022]  1     0 BB20                  0      0    [04F..050)
 BB23 [0023]  3     0 BB19,BB21,BB22        0           [05B..05C)        (finret)    H0   }           i rare 
 -----------------------------------------------------------------------------------------------------------------------------------------
 
-Decided to reverse conditional branch at block BB04 branch to BB06 since it falls into a rarely run block
-fgFindInsertPoint(regionIndex=1, putInTryRegion=true, startBlk=BB02, endBlk=BB08, nearBlk=BB06, jumpBlk=BB00, runRarely=true)
-Relocated rarely run block BB05 by reversing conditional jump at BB04
-Rethreading STMT00034
-Relocated block [BB05..BB05] inserted after BB06
-Block BB05 ended with a BBJ_NONE, Changed to an unconditional jump to BB06
-New Basic Block BB24 [0039] created.
-Setting edge weights for BB06 -> BB24 to [0 .. 3.402823e+38]
-Added an unconditional jump to BB07 after block BB06
-
-After this change in fgReorderBlocks the BB graph is:
------------------------------------------------------------------------------------------------------------------------------------------
-BBnum BBid ref try hnd preds           weight   IBC  lp [IL range]     [jump]      [EH region]         [flags]
------------------------------------------------------------------------------------------------------------------------------------------
-BB01 [0000]  1                             1           [000..00D)                                     i 
-BB02 [0001]  1  0    BB01                  1    100    [00D..025)-> BB05 ( cond ) T0      try {       keep i try idxlen nullcheck IBC 
-BB03 [0014]  1  0    BB02                  0.50  50    [00D..00E)-> BB05 ( cond ) T0                  i IBC 
-BB04 [0015]  1  0    BB03                  0.50  50    [00D..00E)-> BB05 ( cond ) T0                  i IBC 
-BB06 [0018]  2  0    BB04,BB05             1           [???..???)-> BB08 ( cond ) T0                  keep internal idxlen 
-BB24 [0039]  1  0    BB06                  0.50        [???..???)-> BB07 (always) T0                  internal 
-BB05 [0016]  3  0    BB02,BB03,BB04        0      0    [00D..00E)-> BB06 (always) T0                  i rare hascall gcsafe IBC 
-BB07 [0002]  1  0    BB24                  0.50        [025..04C)                 T0      }           i idxlen 
-BB08 [0028]  2       BB06,BB07             1           [04C..04F)-> BB12 ( cond )                     keep i cfb 
-BB09 [0029]  1       BB08                  0.50  50    [04F..05B)-> BB11 ( cond )                     i nullcheck IBC 
-BB10 [0031]  1       BB09                  0.50  50    [04F..050)-> BB12 (always)                     i IBC 
-BB11 [0032]  1       BB09                  0      0    [04F..050)                                     i rare hascall gcsafe IBC 
-BB12 [0007]  3       BB08,BB10,BB11        1           [05B..05F)-> BB18 ( cond )                     i 
-BB13 [0008]  1       BB12                  0.50        [05F..06B)-> BB15 ( cond )                     i hascall new[] 
-BB14 [0038]  1       BB13                  0.25        [???..???)-> BB16 (always)                     i 
-BB15 [0037]  1       BB13                  0.25        [???..???)                                     i 
-BB16 [0035]  2       BB14,BB15             0.50        [06B..07B)-> BB18 ( cond )                     i hascall new[] 
-BB17 [0009]  1       BB16                  0.50        [07B..09B)                                     i hascall gcsafe nullcheck 
-BB18 [0010]  3       BB12,BB16,BB17        1           [09B..09D)        (return)                     i 
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ funclets follow
-BB19 [0004]  1     0                       0           [04C..04F)-> BB23 ( cond )    H0 F fault {     keep i rare flet 
-BB20 [0005]  1     0 BB19                  0      0    [04F..05B)-> BB22 ( cond )    H0               i rare nullcheck IBC 
-BB21 [0021]  1     0 BB20                  0      0    [04F..050)-> BB23 (always)    H0               i rare IBC 
-BB22 [0022]  1     0 BB20                  0      0    [04F..050)                    H0               i rare hascall gcsafe IBC 
-BB23 [0023]  3     0 BB19,BB21,BB22        0           [05B..05C)        (finret)    H0   }           i rare 
------------------------------------------------------------------------------------------------------------------------------------------
-
-Decided to straighten unconditional branch at block BB24 branch to BB07 since it is succeeded by a rarely run block
-fgFindInsertPoint(regionIndex=1, putInTryRegion=true, startBlk=BB02, endBlk=BB08, nearBlk=BB07, jumpBlk=BB00, runRarely=true)
-Could not relocate block BB05
 Decided to straighten unconditional branch at block BB10 branch to BB12 since it is succeeded by a rarely run block
 Relocated rarely run block BB11
 Relocated block [BB11..BB11] inserted after BB18
@@ -85,13 +41,12 @@ After this change in fgReorderBlocks the BB graph is:
 BBnum BBid ref try hnd preds           weight   IBC  lp [IL range]     [jump]      [EH region]         [flags]
 -----------------------------------------------------------------------------------------------------------------------------------------
 BB01 [0000]  1                             1           [000..00D)                                     i 
-BB02 [0001]  1  0    BB01                  1    100    [00D..025)-> BB05 ( cond ) T0      try {       keep i try idxlen nullcheck IBC 
-BB03 [0014]  1  0    BB02                  0.50  50    [00D..00E)-> BB05 ( cond ) T0                  i IBC 
-BB04 [0015]  1  0    BB03                  0.50  50    [00D..00E)-> BB05 ( cond ) T0                  i IBC 
+BB02 [0001]  1  0    BB01                  1           [00D..025)-> BB05 ( cond ) T0      try {       keep i try idxlen nullcheck 
+BB03 [0014]  1  0    BB02                  0.50        [00D..00E)-> BB05 ( cond ) T0                  i 
+BB04 [0015]  1  0    BB03                  0.50        [00D..00E)-> BB06 ( cond ) T0                  i 
+BB05 [0016]  3  0    BB02,BB03,BB04        0.50        [00D..00E)                 T0                  i hascall gcsafe 
 BB06 [0018]  2  0    BB04,BB05             1           [???..???)-> BB08 ( cond ) T0                  keep internal idxlen 
-BB24 [0039]  1  0    BB06                  0.50        [???..???)-> BB07 (always) T0                  internal 
-BB05 [0016]  3  0    BB02,BB03,BB04        0      0    [00D..00E)-> BB06 (always) T0                  i rare hascall gcsafe IBC 
-BB07 [0002]  1  0    BB24                  0.50        [025..04C)                 T0      }           i idxlen 
+BB07 [0002]  1  0    BB06                  0.50        [025..04C)                 T0      }           i idxlen 
 BB08 [0028]  2       BB06,BB07             1           [04C..04F)-> BB12 ( cond )                     keep i cfb 
 BB09 [0029]  1       BB08                  0.50  50    [04F..05B)-> BB11 ( cond )                     i nullcheck IBC 
 BB10 [0031]  1       BB09                  0.50  50    [04F..050)                                     i IBC

@jakobbotsch
Copy link
Member

I don't see many calls to SpinLock.Enter in framework sources beyond this one from ConfigurableArrayPool. It seems plausible that whether or not we will end up with PGO data for this method depends on the exact patterns of contention that occur during the training scenarios.

@jakobbotsch
Copy link
Member

The data we have in the base .mibc is extremely sparse, much too little for us to be basing any decisions off of. We should probably apply a threshold on the dotnet-pgo side when we merge it to StandardOptimizationData.mibc.

    {
      "Method": "[S.P.CoreLib]System.Threading.SpinLock.Enter(bool&)",
      "InstrumentationData": [
        {
          "ILOffset": 0,
          "InstrumentationKind": "NumRuns",
          "Other": 2
        },
        {
          "ILOffset": 13,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 46,
          "Data": 0
        },
        {
          "ILOffset": 27,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 46,
          "Data": 0
        },
        {
          "ILOffset": 27,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 54,
          "Data": 2
        },
        {
          "ILOffset": 54,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 0,
          "Data": 2
        }
      ]
    },

Considering that we do get a good performance gain here though we should probably think about how we can get it via a synthetic profile.

@AndyAyersMS
Copy link
Member

Note that incremental builds with new PGO data does not work in the backwards direction, so you may need a clean build.

Yeah, that's what I was doing. I have two enlistments, both built clean: c9f80fd (base) and a38b50c (diff) -- these are the endpoints from the original report.

a38b50c Remove some unnecessary fixed blocks (#71317) -- (my diff)
aa06c89 Improve allocations and startup in HashAlgorithmNames
d00415b Move helpers for marshaller shape into separate class (#71344)
d99caf1 Make sure we emit diagnostics for "unsupported" marshalling for downlevel platforms (#71342)
601ef26 Simplify "MakeTmpArgNode" (#71019)
5be7092 Argument tests for CreateJsonTypeInfo and CreateJsonPropertyInfo (#71324)
6f7064d JIT: Fix local assertion prop for small locals (#71336)
c9e325d Fix URL to runtime-assets in Capability.Security.cs (#71334)
8fe39db Allow more permissive forward subbing into call arguments (#71161) -- (your diff)
c6584b4 Bump Helix workitem timeout for iOS Device perf scenarios (#71330)
56e4c03 Add annotations for ref fields to public API surface (#71265)
d851aa6 Update dependencies from dotnet/runtime dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness dotnet/emsdk dotnet/roslyn-analyzers (#71128) -- (pgo update)
f8d4cbd Apply Auto layout to large internal structs in System.Text.Json (#71329)
0c473ab Add Regex span startat overloads (#71228)
872fef0 JIT: don't try to shorten an already short arm64 adr (#71306)
475858d [main] Update dependencies from dotnet/linker (#71255)
a052819 Avoid some unnecessary unsigned -> signed casts (#70884)
e8e1451 Support PUTARG_SPLIT(STRUCT LCL_VAR/LCL_FLD) on ARM/64 (#70861)
198b8bf Use Array.Resize in data storages (#71302)
52f4e7c Skip export without EntryPoints from linker script (#71307) -- (your base)
c9f80fd More use of SequenceEqual, IndexOf, IndexOfAnyExcept, etc. (#71278) -- (my base)

Comparing codegen, it looks like I'm on a device with LSE and you're not?

@AndyAyersMS
Copy link
Member

In both my base and diff we find profile data forSpinLock::Enter though it is slightly different.

;;; base

*************** Inline @[000018] Starting PHASE Profile incorporation
Have static profile data: 5 schema records (schema at 0000018CDAEDDAD0, data at 0000018CDAEDDAC0)
Profile summary: 2 runs, 0 block probes, 4 edge probes, 0 class profiles, 0 method profiles, 0 other records

Reconstructing block counts from sparse edge instrumentation
... adding known edge BB15 -> BB17: weight 0
... adding known edge BB16 -> BB17: weight 0
... adding known edge BB16 -> BB18: weight 2
... adding known edge BB18 -> BB14: weight 2

;;; diff

*************** Inline @[000018] Starting PHASE Profile incorporation
Have static profile data: 4 schema records (schema at 000001E7A54ED290, data at 000001E7A54ED280)
Profile summary: 1 runs, 0 block probes, 4 edge probes, 0 class profiles, 0 method profiles, 0 other records

Reconstructing block counts from sparse edge instrumentation
... adding known edge BB15 -> BB17: weight 0
... adding known edge BB16 -> BB17: weight 0
... adding known edge BB16 -> BB18: weight 1
... adding known edge BB18 -> BB14: weight 1

@jakobbotsch
Copy link
Member

Comparing codegen, it looks like I'm on a device with LSE and you're not?

Yes, this is from a Raspberry Pi 400, it does not have LSE.

It is pretty odd that we are ending up with different profile data, let's try to compare the .mibc files from our builds...

@AndyAyersMS
Copy link
Member

Seems like I'm looking at windows, Jakob at ubuntu. The regression was on ubuntu.

@AndyAyersMS
Copy link
Member

Now running locally on ubuntu/ampere. I can repro the slowdown with my base/diff builds (per above).

BenchmarkDotNet=v0.13.1.1786-nightly, OS=ubuntu 22.04
Unknown processor
.NET SDK=7.0.100-preview.5.22307.18
[Host] : .NET 7.0.0 (7.0.22.30112), Arm64 RyuJIT
Job-MPUYMD : .NET 7.0.0 (42.42.42.42424), Arm64 RyuJIT
Job-KVUABQ : .NET 7.0.0 (42.42.42.42424), Arm64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 IterationTime=250.0000 ms MaxIterationCount=20
MinIterationCount=15 WarmupCount=1

Method Job Toolchain RentalSize ManipulateArray Async UseSharedPool Mean Error StdDev Median Min Max Ratio Allocated Alloc Ratio
MultipleSerial Job-MPUYMD /base-rel/corerun 4096 False False False 352.0 ns 1.82 ns 1.70 ns 353.2 ns 349.1 ns 353.8 ns 1.00 - NA
MultipleSerial Job-KVUABQ /diff-rel/corerun 4096 False False False 379.6 ns 1.63 ns 1.52 ns 379.0 ns 377.5 ns 382.4 ns 1.08 - NA

Looking at dumps, they look similar to what @jakobbotsch sees on the Rpi4, there is no pgo data for 'SpinLock:Enter`

Basic block list for 'SpinLock:Enter(byref):this'

-----------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd                 weight    lp [IL range]     [jump]      [EH region]         [flags]
-----------------------------------------------------------------------------------------------------------------------------------------
BB14 [0013]  1                             1       [000..00D)-> BB17 ( cond )                     
BB15 [0014]  1                             1       [00D..01B)-> BB17 ( cond )                     
BB16 [0015]  1                             1       [01B..02E)-> BB18 ( cond )                     
BB17 [0016]  3                             1       [02E..036)                                     
BB18 [0017]  2                             1       [036..037)        (return)                     
-----------------------------------------------------------------------------------------------------------------------------------------

*************** Inline @[000018] Starting PHASE Pre-import

*************** Inline @[000018] Finishing PHASE Pre-import

*************** Inline @[000018] Starting PHASE Profile incorporation
BBOPT set, but no profile data available (hr=80004001)

and this leads to code layout diffs.

We compare a bit more favorably to 6.0 though the numbers for 6.0 bounce around quite a bit.

BenchmarkDotNet=v0.13.1.1786-nightly, OS=ubuntu 22.04
Unknown processor
.NET SDK=7.0.100-rc.2.22417.1
[Host] : .NET 7.0.0 (7.0.22.41112), Arm64 RyuJIT
Job-RDYTPA : .NET 6.0.7 (6.0.722.32202), Arm64 RyuJIT
Job-DKDWDO : .NET 7.0.0 (7.0.22.41112), Arm64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 IterationTime=250.0000 ms MaxIterationCount=20
MinIterationCount=15 WarmupCount=1

Method Job Runtime Toolchain RentalSize ManipulateArray Async UseSharedPool Mean Error StdDev Median Min Max Ratio Allocated Alloc Ratio
MultipleSerial Job-RDYTPA .NET 6.0 net6.0 4096 False False False 368.8 ns 1.38 ns 1.29 ns 369.2 ns 367.2 ns 370.7 ns 1.00 - NA
MultipleSerial Job-DKDWDO .NET 7.0 net7.0 4096 False False False 379.4 ns 1.91 ns 1.79 ns 380.5 ns 376.4 ns 381.3 ns 1.03 - NA

@AndyAyersMS
Copy link
Member

Ampere codegen for the various Rent and Return methods

rr.ampere.diff.dasm.txt
rr.ampere.base.dasm.txt

There is possibly a PGO update coming into 7.0 that might fix this, if not we are just going to have to live with this. As noted above, vs 6.0 we don't look quite as bad.

@jeffhandley
Copy link
Member

There is possibly a PGO update coming into 7.0 that might fix this, if not we are just going to have to live with this. As noted above, vs 6.0 we don't look quite as bad.

@AndyAyersMS Has the PGO update come through, and/or should we go ahead and close this?

@AndyAyersMS
Copy link
Member

We had a PGO update on 8/17 just after the RC1 snap that got backported: #73973

But nothing much seems to have changed for these two tests (note the lab is measuring main). Maybe a slight improvement in the "single parallel" results.

newplot - 2022-09-12T104331 245
newplot - 2022-09-12T104721 371

I don't think we can do anything about this regression for .NET 7. But the underlying issue of very thin PGO data remains, so we should move this to 8.0.

Also, looking a bit more broadly...

All the above regressions have been solved except for System.Buffers.Tests.RentReturnArrayPoolTests MultipleSerial and SingleParallel

@dakersnar I still see more or less all the tests here as regressing -- did "solved" mean that these regressions were deemed necessary somehow?

@dakersnar
Copy link
Contributor

@AndyAyersMS Taking a second look I think I was mistaken. I was referring to the larger July 3rd regression, but this issue is actually referring to the smaller regression on June 27th, which you are correct has not been solved.
image

@jeffhandley
Copy link
Member

There is possibly a PGO update coming into 7.0 that might fix this, if not we are just going to have to live with this. As noted above, vs 6.0 we don't look quite as bad.

Thank you, @AndyAyersMS and @dakersnar. Per Andy's comment above then, we will accept this regression since we are still in good shape for 6.0 vs. 7.0.

@ghost ghost locked as resolved and limited conversation to collaborators Oct 14, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dakersnar dakersnar

Labels
arch-arm64 area-System.Buffers PGO runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
7 participants
@marek-safar @jeffhandley @jakobbotsch @jeffschwMSFT @AndyAyersMS @dakersnar @mrsharm