.NET 7.0 RC2 Microbenchmarks Performance Study Report #77150
Comments
jozkee
added
area-Meta
tenet-performance
|
Tagging subscribers to this area: @dotnet/area-meta Issue DetailsDataThis time we have covered following configurations comparing .NET 7.0 RC2 vs. .NET 6.0:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL. This would not be possible without the help from: @adamsitnik, @brianrob, @carlossanlop, @dakersnar, @janvorli, @jeffhandley, @cincuranet, @tannergooding, and @wfurt who contributed their results and time. An addtional thank you to @dakersnar and @EgorBo for shadowing the creation of this report and helping with the vetting of the results! The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them! Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. RC1 report can be found here. ImprovementsWe will be analyzing improvements more thoroughly in the upcoming .NET 6 vs .NET 7 GA report. This report is focused on regressions. RegressionsBy Design
Investigation In Progress
Noise, Flaky or MultimodalThe following benchmarks showed up in the report generated by the tool, but were not actual regressions:
StatisticsTotal: 86862 Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
|
|
Hoping BigInteger performance improves. C# getting smoked by python computing pi digits is no bueno. https://programming-language-benchmarks.vercel.app/problem/pidigits |
|
Thank you, @jjxtra -- it's helpful to know that BigInteger performance is important to you. We've been accumulating issues around BigInteger and we've been thinking about what it might look like to reimplement it for better performance and reliability. That effort isn't going to be committed for .NET 8, but it is on our radar for the future. It would be helpful data if you could add a comment to #41495 describing your scenarios and how BigInteger's performance affects you. Thanks again! |
|
To be honest, it's probably not impacting anyone severely outside of either: 1] Benchmarks, which give C# a sour taste As I mentioned above, someone seeing a benchmark where python beats C# would be puzzling |
|
It's worth noting that just looking at "raw numbers" is also often not sufficient. You often have to compare the actual implementations, considering where devs have diverged algorithms or done customization as well. In the case of using There are certainly cases where we could/should improve the performance of |
|
I don't know about that site, but the CLBG which it is based on has two categories -- one for the big integer abstractions, and one for anything goes. On the latter, C# is faster than most. |
|
As always its great when community members help improve our submissions in both sites.. |
Very interesting, thanks for the link. |
|
Now I look at the impls there they are all using the GNU arbitrary precision library, so not too interesting perhaps from a .NET point of view. Anyway -- if this stuff interests you perhaps you'd like to look further at what could be done to make things faster, assuming it didn't impact other scenarios. |
Data
This time we have covered following configurations comparing .NET 7.0 RC2 vs. .NET 6.0:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @adamsitnik, @brianrob, @carlossanlop, @dakersnar, @janvorli, @jeffhandley, @cincuranet, @tannergooding, and @wfurt who contributed their results and time. An addtional thank you to @dakersnar and @EgorBo for shadowing the creation of this report and helping with the vetting of the results!
The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. RC1 report can be found here.
Improvements
We will be analyzing improvements more thoroughly in the upcoming .NET 6 vs .NET 7 GA report. This report is focused on regressions.
Regressions
By Design
Benchstone.BenchI.Array2.TestBenchstone.MDBenchI.MDMulMatrix.TestSystem.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 512),System.Collections.Concurrent.IsEmpty<Int32>.Dictionary(Size: 512)System.Collections.IndexerSet<String>.Array(Size: 512)System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False)),System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False)),System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False)),System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt"),System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf")System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]),System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]),System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 0, 0]),System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 1, 1])System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits),System.Numerics.Tests.Perf_BigInteger.Add(arguments: 65536,65536 bits)System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark,System.Numerics.Tests.Perf_Quaternion.EqualsBenchmarkSystem.Numerics.Tests.Perf_VectorConvert.Convert_double_long:System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes.DeriveBytesSystem.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?m)^Sherlock Holmes|Sherlock Holmes$", Options: Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: None)Investigation In Progress
Benchmark.GetChildKeysTests.AddChainedConfigurationEmptyFractalPerf.Launch.TestPerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod,System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024],System.Text.Tests.Perf_Encoding.GetEncoder*System.Collections.Sort<BigStruct>.Array_ComparerClass(Size: 512),System.Collections.Sort<BigStruct>.Array_ComparerStruct(Size: 512),System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512),System.Collections.Sort<BigStruct>.LinqQuery(Size: 512).System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateGetPositionArraySystem.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: List)System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark,System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark,System.Numerics.Tests.Perf_Matrix4x4.Transpose,System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark,System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmarkSystem.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True),System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)System.Tests.Perf_String.CtorCharCount(size: 1),System.Tests.Perf_String.CtorCharCount(size: 10),System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8),System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000),System.Tests.Perf_String.Trim(s: " Test")System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic),System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)Noise, Flaky or Multimodal
The following benchmarks showed up in the report generated by the tool, but were not actual regressions:
System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits)System.Numerics.Tests.Perf_Matrix4x4.LerpBenchmarkSystem.Numerics.Tests.Perf_VectorOf<*>.*Statistics
Total: 86862
Same: 55.57 %
Slower: 7.85 %
Faster: 23.65 %
Noise: 12.88 %
Unknown: 0.04 %
Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
The text was updated successfully, but these errors were encountered: