[Perf] Linux/arm64: 8 Regressions on 5/2/2023 12:51:17 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a87fc1e18e53c03a17ae9105ad679d7906ae9ec
Compare	4ead8078eef158f466b693b2a395b811aa184a1c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ParseSpan - Duration of single invocation	17.56 ns	19.26 ns	1.10	0.04	False
ParseSpan - Duration of single invocation	39.47 ns	49.64 ns	1.26	0.03	False
Parse - Duration of single invocation	17.58 ns	19.47 ns	1.11	0.05	False
Parse - Duration of single invocation	40.01 ns	49.66 ns	1.24	0.01	False
TryParse - Duration of single invocation	16.09 ns	17.93 ns	1.11	0.03	False
TryParse - Duration of single invocation	37.71 ns	47.77 ns	1.27	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.25954464619035 > 18.379914050701448.
IsChangePoint: Marked as a change because one of 1/20/2023 9:27:13 PM, 4/17/2023 8:43:19 PM, 4/25/2023 6:46:56 PM, 5/2/2023 4:25:00 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -55.79028376666772 (T) = (0 -19.410288398403164) / Math.Sqrt((0.006469414607157292 / (8)) + (0.009571357075323483 / (26))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (26) - 2, .025) and -0.10938598557770782 = (17.496424734710654 - 19.410288398403164) / 17.496424734710654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.636308527704614 > 41.33872440282811.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -72.13009480196413 (T) = (0 -49.04932121137935) / Math.Sqrt((0.0419080036788398 / (9)) + (0.33378156761405603 / (25))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (25) - 2, .025) and -0.24585582548390955 = (39.369981829420624 - 49.04932121137935) / 39.369981829420624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.465899363267813 > 18.408046655410278.
IsChangePoint: Marked as a change because one of 1/20/2023 9:27:13 PM, 4/17/2023 8:43:19 PM, 4/25/2023 6:46:56 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -39.03117029057938 (T) = (0 -19.464571075228275) / Math.Sqrt((0.021006366240696548 / (9)) + (0.0074437336385664975 / (24))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (24) - 2, .025) and -0.1149677064331307 = (17.457520036608923 - 19.464571075228275) / 17.457520036608923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.658462108864946 > 41.51664484601211.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -42.94427560659019 (T) = (0 -48.99676693432232) / Math.Sqrt((0.3151486724296729 / (9)) + (0.27912164713415927 / (24))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (24) - 2, .025) and -0.23350004417073247 = (39.721739099946504 - 48.99676693432232) / 39.721739099946504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.TryParse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.926859551449407 > 16.89409475926158.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/25/2023 6:46:56 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -26.90140033275374 (T) = (0 -17.881438958795066) / Math.Sqrt((0.007381033987994321 / (8)) + (0.08182121054366025 / (24))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (24) - 2, .025) and -0.10989862492361012 = (16.110875855915015 - 17.881438958795066) / 16.110875855915015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.77115894217126 > 39.588073334174716.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 5/2/2023 12:51:17 PM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -98.10269115708127 (T) = (0 -47.86814104317204) / Math.Sqrt((0.05448436269928998 / (9)) + (0.11713812496940915 / (24))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (24) - 2, .025) and -0.2727608913757057 = (37.60968880135229 - 47.86814104317204) / 37.60968880135229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a87fc1e18e53c03a17ae9105ad679d7906ae9ec
Compare	4ead8078eef158f466b693b2a395b811aa184a1c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Divide - Duration of single invocation	10.23 ns	13.47 ns	1.32	0.42	False
Remainder - Duration of single invocation	8.56 ns	11.72 ns	1.37	0.21	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.Divide(arguments: 16,8 bits)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.468836175653925 > 12.032495606902273.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 4:25:00 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -11.46482711577723 (T) = (0 -13.516268937131917) / Math.Sqrt((0.2555043325017234 / (8)) + (0.15539304440412818 / (25))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (25) - 2, .025) and -0.1985859720095326 = (11.276845593704662 - 13.516268937131917) / 11.276845593704662 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_BigInteger.Remainder(arguments: 16,8 bits)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.720990070391744 > 9.006522496559263.
IsChangePoint: Marked as a change because one of 5/2/2023 4:25:00 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -16.020109436859666 (T) = (0 -11.682054332822247) / Math.Sqrt((0.3162785154714892 / (8)) + (0.1296505720677508 / (25))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (25) - 2, .025) and -0.40845497222083776 = (8.294233442480664 - 11.682054332822247) / 8.294233442480664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[EgorBo]

Could be dotnet/runtime#85620 and is expected to be fixed by dotnet/runtime#86246

[kunalspathak]

dotnet/runtime#85987