[Perf] Linux/x64: 1 Regression on 8/20/2023 5:37:20 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	a429b0ca4fbb2ed7014815792f1cf1b5939eb567
Compare	d74b52492d85a3a250654d670942e35f9f0be2e8
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IndexOfAnyTwoValues - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	1.56 ns	2.65 ns	1.70	0.33	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*'

Payloads

Baseline
Compare

System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 4)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.654301425878747 > 1.8619193806192755.
IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM.
IsRegressionStdDev: Marked as regression because -5.451648309262017 (T) = (0 -2.1929101539938283) / Math.Sqrt((0.05550951201037635 / (8)) + (0.025818483466123296 / (27))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (27) - 2, .025) and -0.28352004260322977 = (1.7085125913158141 - 2.1929101539938283) / 1.7085125913158141 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[cincuranet]

The diff points to #88394 cc @MihaZupan. But I'm not 100% convinced.

[MihaZupan]

That PR is adding new APIs, but they're not consumed anywhere yet, so I doubt it's the cause.

[cincuranet]

The first spike points to d267844...374b111. That seems to be hardly related as well.

[cincuranet]

Also seems to be only Linux/Ubuntu 22.04 x64. Others are unaffected.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-memory
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	a429b0ca4fbb2ed7014815792f1cf1b5939eb567
Compare	d74b52492d85a3a250654d670942e35f9f0be2e8
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IndexOfAnyTwoValues - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	1.56 ns	2.65 ns	1.70	0.33	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*'

Payloads

Baseline
Compare

System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 4)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.654301425878747 > 1.8619193806192755.
IsChangePoint: Marked as a change because one of 8/19/2023 11:06:43 PM, 8/28/2023 11:06:53 PM falls between 8/19/2023 11:06:43 PM and 8/28/2023 11:06:53 PM.
IsRegressionStdDev: Marked as regression because -5.451648309262017 (T) = (0 -2.1929101539938283) / Math.Sqrt((0.05550951201037635 / (8)) + (0.025818483466123296 / (27))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (8) + (27) - 2, .025) and -0.28352004260322977 = (1.7085125913158141 - 2.1929101539938283) / 1.7085125913158141 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Memory, os-linux, arch-x64, untriaged, runtime-coreclr, needs-area-label
Milestone:	-

[adamsitnik]

That PR is adding new APIs, but they're not consumed anywhere yet, so I doubt it's the cause.

Adding new methods can change the alignment of other methods. It would be great to compare the disassembly to see if anything has actually changed.

[tannergooding]

This is a 1 nanosecond change for a scenario taking in 4 inputs in an API that has undergone multiple other improvements balancing APIs .

I'm closing this as "by design" given the regression is so small, that there is no clear indicator of what caused the regression, and that such small timings can be influenced by almost any part of the system.