Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 48 Regressions on 2/27/2024 9:28:58 PM #99315

Open
performanceautofiler bot opened this issue Mar 5, 2024 · 13 comments
Open

[Perf] Linux/x64: 48 Regressions on 2/27/2024 9:28:58 PM #99315

performanceautofiler bot opened this issue Mar 5, 2024 · 13 comments
Assignees
@amanasifkhalid
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
10.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 5, 2024 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.63 μs 1.94 μs 1.19 0.20 False
183.62 ns 227.43 ns 1.24 0.25 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;Int16&gt;*'

Payloads

Baseline
Compare

System.Collections.Perf_Frozen<Int16>.TryGetValue_True(Count: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Perf_Frozen<Int16>.TryGetValue_True(Count: 64)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
13.49 μs 16.21 μs 1.20 0.15 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsKeyFalse<Int32, Int32>.ImmutableDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
12.89 μs 14.69 μs 1.14 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsTrue<Int32>.ImmutableHashSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
18.12 ms 28.51 ms 1.57 0.01 False
18.10 ms 28.55 ms 1.58 0.11 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: None)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEachNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
922.25 ns 1.37 μs 1.48 0.02 False
1.60 μs 2.03 μs 1.27 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateForEachNonGeneric<Int32>.ArrayList(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.IterateForEachNonGeneric<Int32>.Queue(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.31 ms 1.61 ms 1.23 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
647.11 ns 736.15 ns 1.14 0.04 False
146.00 ns 176.60 ns 1.21 0.07 False
58.07 ns 67.16 ns 1.16 0.03 False
1.82 ns 4.21 ns 2.32 0.06 False
125.66 ns 139.97 ns 1.11 0.01 False
146.65 ns 176.25 ns 1.20 0.12 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

System.Linq.Tests.Perf_Enumerable.Zip(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: Array)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.ElementAt(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.ElementAt(input: IList)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
21.07 ns 28.86 ns 1.37 0.10 False
20.53 ns 28.63 ns 1.39 0.11 False
21.14 ns 29.06 ns 1.37 0.12 True

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.30 μs 2.61 μs 1.13 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateForEach<String>.ConcurrentStack(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Char

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.35 ns 15.25 ns 2.08 0.04 True
8.80 ns 14.15 ns 1.61 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Char*'

Payloads

Baseline
Compare

System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
65.29 ns 72.09 ns 1.10 0.01 False
66.35 ns 71.66 ns 1.08 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Mar 5, 2024
@performanceautofiler PerformanceAutoFiler
Copy link
Author

performanceautofiler bot commented Mar 5, 2024 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.86 ns 12.26 ns 1.13 0.23 False
8.32 ns 9.43 ns 1.13 0.14 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

ETL Files

Histogram

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.86 ns 13.90 ns 1.28 0.05 False
7.09 ns 8.62 ns 1.22 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64Hex(value: FFFFFFFFFFFFFFFF)

ETL Files

Histogram

JIT Disasms

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt32Hex(value: FFFFFFFFFFFFFFFF)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
838.47 ns 1.03 μs 1.23 0.08 False
2.17 μs 2.32 μs 1.07 0.10 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.TryGetValueFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.TryGetValueFalse<Int32, Int32>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEachNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.60 μs 2.26 μs 1.42 0.02 False
921.68 ns 1.36 μs 1.48 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateForEachNonGeneric<String>.Queue(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.IterateForEachNonGeneric<String>.ArrayList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
45.96 ns 49.93 ns 1.09 0.29 False
44.79 ns 50.58 ns 1.13 0.19 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

System.Tests.Perf_Double.TryParse(value: "12345")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Double.Parse(value: "12345")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
66.57 ns 73.98 ns 1.11 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,�2020,512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
23.33 μs 24.91 μs 1.07 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>.JsonNet_

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_DateTime

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
24.57 ns 29.85 ns 1.21 0.15 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_DateTime*'

Payloads

Baseline
Compare

System.Tests.Perf_DateTime.ToString(format: "r")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Logging.ScopesOverheadBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.53 ns 4.81 ns 1.36 0.07 False
9.49 ns 11.28 ns 1.19 0.07 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Logging.ScopesOverheadBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.Logging.ScopesOverheadBenchmark.FilteredByLevel(HasISupportLoggingScopeLogger: False, CaptureScopes: False)

ETL Files

Histogram

JIT Disasms

Microsoft.Extensions.Logging.ScopesOverheadBenchmark.FilteredByLevel_InsideScope(HasISupportLoggingScopeLogger: False, CaptureScopes: False)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.40 ns 17.48 ns 1.14 0.16 False
13.23 ns 15.22 ns 1.15 0.11 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

System.Tests.Perf_String.Trim(s: " Te st ")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Trim(s: " Test")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.07 ns 18.62 ns 1.32 0.45 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

System.Tests.Perf_Int32.Parse(value: "-2147483648")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in BenchmarksGame.FannkuchRedux_5

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
22.02 ms 26.34 ms 1.20 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BenchmarksGame.FannkuchRedux_5*'

Payloads

Baseline
Compare

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
464.41 ns 563.02 ns 1.21 0.01 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.IndexerSet<Int32>.List(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler PerformanceAutoFiler
Copy link
Author

performanceautofiler bot commented Mar 5, 2024 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Struct.GSeq

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.35 μs 12.04 μs 1.16 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Struct.GSeq*'

Payloads

Baseline
Compare

Struct.GSeq.FilterSkipMapSum

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
488.60 μs 541.90 μs 1.11 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsFalse<String>.LinkedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_LengthBucketsFrozenDictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.13 μs 1.24 μs 1.10 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_LengthBucketsFrozenDictionary*'

Payloads

Baseline
Compare

System.Collections.Perf_LengthBucketsFrozenDictionary.ToFrozenDictionary(Count: 100, ItemsPerBucket: 1)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
47.00 ns 52.04 ns 1.11 0.18 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*'

Payloads

Baseline
Compare

System.Tests.Perf_Single.TryParse(value: "12345")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.49 ms 1.74 ms 1.17 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: NonBacktracking)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot mentioned this issue Mar 5, 2024
Closed
@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues Mar 5, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Mar 5, 2024
@DrewScoggins
Copy link
Member

@AndyAyersMS this change looks like it could be the impact #98324

@DrewScoggins
Copy link
Member

DrewScoggins commented Mar 5, 2024 •
edited
Loading

Regressions:
Linux x64 - dotnet/perf-autofiling-issues#30417
Windows x64 - dotnet/perf-autofiling-issues#30479
Windows x64 - dotnet/perf-autofiling-issues#30511

Improvements:
dotnet/perf-autofiling-issues#30426
dotnet/perf-autofiling-issues#30520

@DrewScoggins DrewScoggins added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Mar 5, 2024
@AndyAyersMS
Copy link
Member

Looks like there are quite a few improvements too (though it could also be #97865)

@AndyAyersMS
Copy link
Member

@AndyAyersMS this change looks like it could be the impact #98324

Or possibly #97865

@jakobbotsch
Copy link
Member

jakobbotsch commented Mar 6, 2024 •
edited
Loading

Report for dotnet/perf-autofiling-issues#30479: https://github.com/jakobbotsch/perf-diff-finder/blob/main/30479/results.md. System.Collections.ContainsKeyTrue<Int32, Int32>.ImmutableDictionary(Size: 512) looks like it could be IV widening related (though the potentially bad diff there looks flowgraph related). The remaining IV widening diffs look good.

Report for dotnet/perf-autofiling-issues#30511: https://github.com/jakobbotsch/perf-diff-finder/blob/main/30511/results.md
Hard to tell exactly what the cause of these is. Many of them have IV widening in them, but most of the diffs look like changes in flowgraph opts. Those could certainly have been induced by IV widening (I saw a number of these kinds of diffs in the original PR).

@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 6, 2024
@ghost
Copy link

ghost commented Mar 6, 2024

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.63 μs 1.94 μs 1.19 0.20 False
183.62 ns 227.43 ns 1.24 0.25 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;Int16&gt;*'

Payloads

Baseline
Compare

System.Collections.Perf_Frozen<Int16>.TryGetValue_True(Count: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Perf_Frozen<Int16>.TryGetValue_True(Count: 64)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
13.49 μs 16.21 μs 1.20 0.15 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsKeyFalse<Int32, Int32>.ImmutableDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
12.89 μs 14.69 μs 1.14 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsTrue<Int32>.ImmutableHashSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
18.12 ms 28.51 ms 1.57 0.01 False
18.10 ms 28.55 ms 1.58 0.11 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: None)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEachNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
922.25 ns 1.37 μs 1.48 0.02 False
1.60 μs 2.03 μs 1.27 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateForEachNonGeneric<Int32>.ArrayList(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.IterateForEachNonGeneric<Int32>.Queue(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.31 ms 1.61 ms 1.23 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
647.11 ns 736.15 ns 1.14 0.04 False
146.00 ns 176.60 ns 1.21 0.07 False
58.07 ns 67.16 ns 1.16 0.03 False
1.82 ns 4.21 ns 2.32 0.06 False
125.66 ns 139.97 ns 1.11 0.01 False
146.65 ns 176.25 ns 1.20 0.12 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

System.Linq.Tests.Perf_Enumerable.Zip(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: Array)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.ElementAt(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.ElementAt(input: IList)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
21.07 ns 28.86 ns 1.37 0.10 False
20.53 ns 28.63 ns 1.39 0.11 False
21.14 ns 29.06 ns 1.37 0.12 True

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.30 μs 2.61 μs 1.13 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateForEach<String>.ConcurrentStack(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Char

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.35 ns 15.25 ns 2.08 0.04 True
8.80 ns 14.15 ns 1.61 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Char*'

Payloads

Baseline
Compare

System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 0cb48f4f2de372118a324be32a3de77340c30f22
Compare 40d1c8975d8c66308f18c04bed41fcd0d3dedf71
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
65.29 ns 72.09 ns 1.10 0.01 False
66.35 ns 71.66 ns 1.08 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, area-CodeGen-coreclr, untriaged, runtime-coreclr, needs-area-label
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Mar 7, 2024

ARM64: dotnet/perf-autofiling-issues#30625

@stephentoub stephentoub mentioned this issue Mar 8, 2024
Merged
@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Mar 12, 2024
@JulieLeeMSFT JulieLeeMSFT added this to the 9.0.0 milestone Mar 12, 2024
@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Mar 12, 2024
@JulieLeeMSFT
Copy link
Member

@jakobbotsch, if it is not from IV widening, please reassign this to @amanasifkhalid.

@jakobbotsch jakobbotsch added the Priority:2 Work that is important, but not critical for the release label May 3, 2024
@jakobbotsch
Copy link
Member

Many of these have moved significantly with recent flowgraph work. There are a few clear regressions:

  • Char_ToUpperInvariant
  • TryParseUInt64Hex
  • System.Collections.IndexerSet.List(Size: 512)

and also a few that improved with some flowgraph changes, then regressed again with other flowgraph changes.

@jakobbotsch
Copy link
Member

A lot of these were fixed or have had several swings around different flowgraph PRs. From what I can see, there are still a few regressions:

  • System.Collections.IterateForEachNonGeneric
  • System.Collections.IndexerSet.List

We won't churn the flowgraph code anymore in .NET 9, so I will move this to .NET 10 and also give it to Aman to check up on these benchmarks as we do flow graph work in .NET 10.

@jakobbotsch jakobbotsch modified the milestones: 9.0.0, 10.0.0 Aug 7, 2024
@jakobbotsch
Copy link
Member

Actually the IndexerSet one might be related to #104655, so that one I should keep track of in .NET 10.

@jakobbotsch jakobbotsch mentioned this issue Aug 8, 2024
Closed
@DrewScoggins DrewScoggins mentioned this issue Aug 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amanasifkhalid amanasifkhalid

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
8 participants
@vcsjones @EgorBo @jakobbotsch @DrewScoggins @jeffschwMSFT @AndyAyersMS @amanasifkhalid @JulieLeeMSFT