[Perf] Windows/arm64: 5 Regressions on 6/8/2023 3:52:42 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare	9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.DictionarySequentialKeys

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryGetValue_17_Int_Int - Duration of single invocation	5.88 ns	7.40 ns	1.26	0.72	False
TryGetValue_3k_Int_Int - Duration of single invocation	5.61 ns	7.64 ns	1.36	0.68	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.DictionarySequentialKeys*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.395976936880073 > 5.7943076452128945.
IsChangePoint: Marked as a change because one of 3/31/2023 7:02:21 PM, 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.695170562714825 (T) = (0 -8.018789895969261) / Math.Sqrt((0.18774556681341403 / (12)) + (1.3012597196780862 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.484735265675485 = (5.40082133249599 - 8.018789895969261) / 5.40082133249599 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.641963550983901 > 5.879509724413551.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -13.113548564576806 (T) = (0 -8.743435290828566) / Math.Sqrt((0.17092856038841 / (12)) + (0.6791435685309158 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.6305911085842962 = (5.362126191415179 - 8.743435290828566) / 5.362126191415179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare	9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryParseDateTimeOffset - Duration of single invocation	5.97 ns	7.64 ns	1.28	0.83	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseDateTimeOffset(value: 12/30/2017 3:45:22 AM -08:00)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.640911329696776 > 6.302415903491371.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.262938294741985 (T) = (0 -9.710078430474258) / Math.Sqrt((0.9973057405368995 / (12)) + (2.751057774974143 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.6836997855265233 = (5.7671079570980295 - 9.710078430474258) / 5.7671079570980295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare	9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Replace_String - Duration of single invocation	73.13 ns	103.03 ns	1.41	0.58	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.02587738588925 > 79.04393196166279.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -6.95275929703907 (T) = (0 -93.03306473253484) / Math.Sqrt((15.875856829350282 / (12)) + (64.28241464298715 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.23016574351153538 = (75.62644726796724 - 93.03306473253484) / 75.62644726796724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare	9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetUInt16 - Duration of single invocation	1.28 μs	1.42 μs	1.11	0.67	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Get.GetUInt16

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4243274439721787 > 1.1817496321292034.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.208603559815226 (T) = (0 -1474.2133425890993) / Math.Sqrt((6162.145617042773 / (12)) + (15447.056533549852 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.25268554569773194 = (1176.8423030442 - 1474.2133425890993) / 1176.8423030442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[AndyAyersMS]

Likely dotnet/runtime#87115

@EgorBo FYI