[Perf] Windows/x64: 6 Regressions in Perf_Vector on 6/20/2024 7:04:47 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	6d838df6888da0060984526ea26960709447f304
Compare	4a7fe654d798a372f5786f026006437444f14f1e
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
TransformByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.02 ns	1.11 ns	64.03	0.38	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*'

System.Numerics.Tests.Perf_Vector2.TransformByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	6d838df6888da0060984526ea26960709447f304
Compare	4a7fe654d798a372f5786f026006437444f14f1e
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Plane

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
TransformByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.27 ns	1.63 ns	6.01	0.36	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*'

System.Numerics.Tests.Perf_Plane.TransformByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	6d838df6888da0060984526ea26960709447f304
Compare	4a7fe654d798a372f5786f026006437444f14f1e
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector4

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
TransformVector2ByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.54 ns	1.62 ns	3.00	0.27	False
TransformVector3ByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.54 ns	1.63 ns	2.99	0.20	False
TransformByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.54 ns	1.63 ns	3.00	0.21	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'

System.Numerics.Tests.Perf_Vector4.TransformVector2ByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_Vector4.TransformVector3ByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_Vector4.TransformByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	6d838df6888da0060984526ea26960709447f304
Compare	4a7fe654d798a372f5786f026006437444f14f1e
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
TransformByQuaternionBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	0.54 ns	1.62 ns	2.99	0.30	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'

System.Numerics.Tests.Perf_Vector3.TransformByQuaternionBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

[DrewScoggins]

Seems to be from #103527

[DrewScoggins]

Regressions:
Windows x64 - dotnet/perf-autofiling-issues#36989
Linux x64 - dotnet/perf-autofiling-issues#36882
Windows Arm64 - dotnet/perf-autofiling-issues#37200, dotnet/perf-autofiling-issues#37137
Linux Arm64 - dotnet/perf-autofiling-issues#37168

[jeffhandley]

@tannergooding / @michaelgsharp - I've marked this as blocking-release for 9.0.0 to ensure we triage this further and determine if we need to make a fix before the release.

[tannergooding]

Each of these is a scenario where we increased the execution time by about 1ns (often from 0ns), so none of these should be a concerns.

Glancing at a few of them, it looks like a case where the benchmarks were bad and were effectively measuring nothing due to constant folding that took place for the previously scalar implementation.

The real world cases (where there is no or only limited constant folding) are now significantly faster. We could notably resolve the difference and ensure the new implementation also fully constant folds, it would just require recognizing FusedMultiplyAdd and Shuffle, but I don't think that's worth the additional effort for .NET 9 and we can handle it in .NET 10 instead

In .NET 10 we should also take a look at improving these benchmarks so they're measuring the real cost and not allowing it to be folded away to nothing.