[Perf] Linux/x64: 3 Regressions on 7/26/2023 12:02:55 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	8f5364695b107f37af899aba3ad1b9f98d577c33
Compare	7ebf9676b4e37c183072f97efaa21e6b4a7d33f3
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IsPrefix_FirstHalf - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	24.88 ns	28.33 ns	1.14	0.16	False
LastIndexOf_Word_NotFound - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	19.16 ns	22.78 ns	1.19	0.22	False
IsSuffix_SecondHalf - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	25.34 ns	28.32 ns	1.12	0.14	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.328953841476547 > 25.98331537698991.
IsChangePoint: Marked as a change because one of 4/26/2023 8:04:03 AM, 5/19/2023 10:30:38 AM, 7/25/2023 7:04:35 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -39.78532408452458 (T) = (0 -28.480675338250755) / Math.Sqrt((0.09244359956646445 / (24)) + (0.07784766894706631 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -0.1499881218548519 = (24.766060446183896 - 28.480675338250755) / 24.766060446183896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.77511804372087 > 20.126789623174854.
IsChangePoint: Marked as a change because one of 5/19/2023 12:51:11 AM, 6/9/2023 2:21:24 AM, 6/20/2023 9:17:03 AM, 7/23/2023 4:13:29 AM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -26.696087393453077 (T) = (0 -22.257811119112844) / Math.Sqrt((0.15607099494786222 / (15)) + (0.05606845123366204 / (26))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (26) - 2, .025) and -0.155301756268404 = (19.265798739028167 - 22.257811119112844) / 19.265798739028167 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.323701576469187 > 26.615446916967073.
IsChangePoint: Marked as a change because one of 5/16/2023 5:40:18 PM, 7/25/2023 7:04:35 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -54.371987492984864 (T) = (0 -28.45550903642124) / Math.Sqrt((0.05707986766085929 / (24)) + (0.016128642990411245 / (17))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (17) - 2, .025) and -0.12386674929799374 = (25.319290791542272 - 28.45550903642124) / 25.319290791542272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	8f5364695b107f37af899aba3ad1b9f98d577c33
Compare	7ebf9676b4e37c183072f97efaa21e6b4a7d33f3
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IsPrefix_FirstHalf - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	24.88 ns	28.33 ns	1.14	0.16	False
LastIndexOf_Word_NotFound - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	19.16 ns	22.78 ns	1.19	0.22	False
IsSuffix_SecondHalf - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	25.34 ns	28.32 ns	1.12	0.14	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.328953841476547 > 25.98331537698991.
IsChangePoint: Marked as a change because one of 4/26/2023 8:04:03 AM, 5/19/2023 10:30:38 AM, 7/25/2023 7:04:35 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -39.78532408452458 (T) = (0 -28.480675338250755) / Math.Sqrt((0.09244359956646445 / (24)) + (0.07784766894706631 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -0.1499881218548519 = (24.766060446183896 - 28.480675338250755) / 24.766060446183896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.77511804372087 > 20.126789623174854.
IsChangePoint: Marked as a change because one of 5/19/2023 12:51:11 AM, 6/9/2023 2:21:24 AM, 6/20/2023 9:17:03 AM, 7/23/2023 4:13:29 AM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -26.696087393453077 (T) = (0 -22.257811119112844) / Math.Sqrt((0.15607099494786222 / (15)) + (0.05606845123366204 / (26))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (26) - 2, .025) and -0.155301756268404 = (19.265798739028167 - 22.257811119112844) / 19.265798739028167 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.323701576469187 > 26.615446916967073.
IsChangePoint: Marked as a change because one of 5/16/2023 5:40:18 PM, 7/25/2023 7:04:35 PM, 7/31/2023 8:02:29 AM falls between 7/22/2023 12:54:19 PM and 7/31/2023 8:02:29 AM.
IsRegressionStdDev: Marked as regression because -54.371987492984864 (T) = (0 -28.45550903642124) / Math.Sqrt((0.05707986766085929 / (24)) + (0.016128642990411245 / (17))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (17) - 2, .025) and -0.12386674929799374 = (25.319290791542272 - 28.45550903642124) / 25.319290791542272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	os-linux, arch-x64, area-CodeGen-coreclr, runtime-coreclr
Milestone:	-

[cincuranet]

The commit range is 764f774...d389ab9 and #89059 looks like a suspect.

[JulieLeeMSFT]

@tannergooding PTAL.

[tannergooding]

Will take a look, but its worth noting these perf lab machines don't have AVX-512

[tannergooding]

This is a very small regression on a subset of hardware/scenarios from using ptest; jcc instead of pcmp; movmsk; cmp; jcc

We saw some perf wins in other scenarios, overall smaller codegen, and better perf on a variety of later machines;

Given that, and the very minimal size of the regression here, it should be within an acceptable range.