Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.1 vs 2.2 comparison using performance repo #193

Closed
adamsitnik opened this issue Dec 14, 2018 · 9 comments
Closed

2.1 vs 2.2 comparison using performance repo #193

adamsitnik opened this issue Dec 14, 2018 · 9 comments

Comments

@adamsitnik
Copy link
Member

adamsitnik commented Dec 14, 2018 •
edited
Loading

In this issue, I am going to post 2.1 vs 2.2 results for the benchmarks we have here using ResultsComparer from #165

Sample command:

git clone https://github.com/dotnet/performance.git
cd performance
.\scripts\benchmarks_ci.py --frameworks netcoreapp2.1 netcoreapp2.2 --architecture x64 --filter *
dotnet run -p .\src\tools\ResultsComparer\ResultsComparer.csproj 
   --base ".\src\benchmarks\micro\bin\Release\netcoreapp2.1\BenchmarkDotNet.Artifacts\results\" 
   --diff ".\src\benchmarks\micro\bin\Release\netcoreapp2.2\BenchmarkDotNet.Artifacts\results\"
   --threshold 10% --noise 1ns
@adamsitnik
Copy link
Member Author

adamsitnik commented Dec 14, 2018 •
edited
Loading

x64 Windows 10 PC

Windows 10.0.17763.134 (1809/October2018Update/Redstone5)
Intel Xeon CPU E5-1650 v4 3.60GHz
BenchmarkDotNet 0.11.3.886-nightly
base: .NET Core 2.1.5 (CoreCLR 4.6.26919.02, CoreFX 4.6.26919.02)
diff: .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04)

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.Memory.Span<Byte>.Clear(Size: 512) 1.60 7.61 12.18 bimodal
System.Collections.ContainsKeyFalse<Int32, Int32>.Dictionary(Size: 512) 1.57 3326.37 5208.86
System.Memory.Span<Byte>.IndexOfValue(Size: 512) 1.45 9.29 13.47 several?
System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512) 1.32 16.73 22.11 several?
System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], opti 1.28 35.70 45.65
Benchstone.BenchF.NewtR.Test 1.27 233146350.00 295478650.00 several?
System.Memory.Span<Byte>.Fill(Size: 512) 1.24 10.51 13.05
System.Collections.IndexerSetReverse<String>.IList(Size: 512) 1.23 1962.91 2410.12
System.Collections.IterateForEach<Int32>.LinkedList(Size: 512) 1.22 1839.53 2250.22 bimodal
System.Collections.Remove<Int32>.Stack(Size: 512) 1.20 964.25 1155.63
System.Collections.IterateForEach<Int32>.ImmutableStack(Size: 512) 1.19 3410.11 4064.06 bimodal
Benchstone.BenchI.NDhrystone.Test 1.19 473228550.00 562947150.00 bimodal
System.Collections.IterateForEach<Int32>.ImmutableQueue(Size: 512) 1.19 3653.33 4332.89
System.Tests.Perf_String.Compare(strings: ["The quick brown fox", "THE QUICK BRO 1.19 30.16 35.76
PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod 1.18 177241.46 209246.39 bimodal
System.Collections.ContainsTrue<String>.ICollection(Size: 512) 1.18 439943.99 518737.09
System.Collections.IterateForEach<Int32>.Queue(Size: 512) 1.18 1445.51 1702.58
System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], opti 1.18 37.14 43.73 bimodal
System.Threading.Tasks.ValueTaskPerfTest.Await_FromCompletedValueTaskSource 1.17 77.10 90.50
System.IO.Tests.Perf_FileStream.WriteAsync(BufferSize: 200000, TotalSize: 200000 1.17 1021418.83 1198154.03 several?
System.Collections.AddGivenSize<Int32>.Dictionary(Size: 512) 1.17 6432.84 7507.43
PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod 1.17 158754.16 185242.74 bimodal
System.Tests.Perf_String.IndexerCheckPathLength 1.16 113.27 131.31
System.Memory.Constructors<String>.SpanImplicitCastFromArray 1.16 12.19 14.11
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 1000) 1.16 27.26 31.55 bimodal
System.Collections.Remove<String>.LinkedList(Size: 512) 1.16 10525.00 12161.91
System.Collections.TryAddGiventSize<Int32>.Dictionary(Count: 512) 1.15 6628.18 7607.06
System.Collections.CtorFromCollection<Int32>.Dictionary(Size: 512) 1.15 7119.89 8158.07
System.Memory.Constructors<String>.SpanFromArray 1.15 11.47 13.14
MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>.Utf8Json_ 1.14 440855.22 504223.99 bimodal
System.Linq.Tests.Perf_Linq.WhereSelect(size: 100, iteration: 1000, wrapType: IE 1.14 1266.44 1448.00 several?
System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "ascii") 1.14 7.63 8.70
System.Linq.Tests.Perf_Linq.WhereSelect(size: 100, iteration: 1000, wrapType: IR 1.14 1270.29 1444.88 several?
System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512) 1.14 30344.87 34487.34
System.Collections.Clear<String>.Array(Size: 512) 1.13 189.96 215.19
System.Linq.Tests.Perf_Linq.WhereSelect(size: 100, iteration: 1000, wrapType: IC 1.13 1274.46 1442.77 several?
System.Collections.IterateFor<Int32>.ImmutableList(Size: 512) 1.13 14452.38 16285.09
System.Collections.Concurrent.IsEmpty<Int32>.Bag(Size: 512) 1.12 12.95 14.54
Faster base/diff Base Median (ns) Diff Median (ns) Modality
Benchstone.BenchI.IniArray.Test 1.63 95278850.00 58464775.00 bimodal
System.Collections.IterateForEach<String>.ConcurrentDictionary(Size: 512) 1.47 25601.38 17403.67
System.Tests.Perf_String.Trim_CharArr(s: "Test", c: [' ', ' ']) 1.47 7.54 5.14
System.Collections.AddDefaultSize<String>.ConcurrentDictionary(Count: 512) 1.36 117452.49 86257.21 bimodal
System.Globalization.Tests.Perf_CultureInfo.GetCurrentCulture 1.33 11.74 8.84
System.Tests.Perf_String.TrimStart_CharArr(s: "Test", c: [' ', ' ']) 1.31 5.80 4.43
MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>.Utf8Json_ 1.31 41850.00 32062.74
System.Collections.CopyTo<Int32>.Array(Size: 2048) 1.30 211.39 162.52
System.Collections.CopyTo<Int32>.ReadOnlyMemory(Size: 2048) 1.29 233.89 181.16 bimodal
System.Tests.Perf_String.PadLeft(n: 18) 1.28 17.77 13.85
System.Collections.CtorDefaultSize<String>.Stack 1.28 16.90 13.23
System.Perf_Convert.ChangeType 1.28 59.28 46.46
System.Tests.Perf_Array.ArrayCopy2D 1.26 335.94 266.61
System.Collections.CopyTo<Int32>.ReadOnlySpan(Size: 2048) 1.25 209.88 167.31
System.Collections.CopyTo<Int32>.Span(Size: 2048) 1.25 208.86 166.61 several?
System.Diagnostics.Perf_Process.StartAndKillDotNetVersion 1.25 5091073.91 4065356.67 bimodal
System.IO.Compression.Brotli.Compress(file: alice29.txt, level: Optimal) 1.23 34373.19 27933.70 several?
System.Collections.IndexerSet<String>.Span(Size: 512) 1.22 360.39 294.97 several?
MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.Utf8Json_ 1.22 829050.00 678900.00
System.Collections.CtorDefaultSize<String>.SortedSet 1.22 15.63 12.81
System.Tests.Perf_String.IndexerCheckLengthHoisting 1.22 38.81 31.86 bimodal
System.Collections.ContainsKeyFalse<String, String>.ConcurrentDictionary(Size: 5 1.21 25500.39 21156.47
System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 1.19 417235.57 349497.69
System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", ol 1.19 36.15 30.46
System.IO.Compression.Brotli.Compress(file: TestDocument.pdf, level: Optimal) 1.18 26168.69 22142.63
System.IO.Tests.Perf_FileStream.Write(BufferSize: 200000, TotalSize: 200000) 1.18 929551.89 790327.91 bimodal
System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch_Multithreading(tot 1.17 20740872.73 17688009.09
System.IO.Compression.Brotli.Compress_WithState(file: TestDocument.pdf, level: F 1.17 80104.38 68550.03
System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' ']) 1.17 15.36 13.18
System.Collections.CopyTo<Int32>.Memory(Size: 2048) 1.16 223.87 192.59
System.Linq.Tests.Perf_Linq.Cast_SameType(size: 100, iteration: 1000, wrapType: 1.16 94.64 81.54 bimodal
MicroBenchmarks.Serializers.Xml_ToStream<SimpleStructWithProperties>.DataContrac 1.16 1164.54 1003.88
System.Tests.Perf_String.ToUpper(s: "TeSt") 1.16 40.33 34.91 several?
MicroBenchmarks.Serializers.Json_ToString<IndexViewModel>.JsonNet_ 1.15 52583.50 45562.09 bimodal
Span.Sorting.QuickSortArray(Size: 512) 1.15 12029.76 10472.06
System.Memory.Span<Int32>.BinarySearch(Size: 512) 1.15 20.81 18.15
System.IO.Tests.Perf_Path.GetTempPath 1.14 414.14 361.70
System.Memory.Span<Int32>.IndexOfValue(Size: 512) 1.14 67.41 58.88
MicroBenchmarks.Serializers.Xml_ToStream<XmlElement>.DataContractSerializer_ 1.14 549.85 480.37 several?
System.Collections.ContainsTrue<String>.HashSet(Size: 512) 1.14 25871.62 22674.49
System.Collections.CtorDefaultSize<String>.SortedList 1.14 32.04 28.16 several?
MicroBenchmarks.Serializers.Json_ToStream<CollectionsOfPrimitives>.Utf8Json_ 1.13 379800.00 334800.00
System.Collections.CtorDefaultSize<Int32>.Dictionary 1.13 11.81 10.43
MicroBenchmarks.Serializers.Xml_ToStream<ClassImplementingIXmlSerialiable>.DataC 1.13 819.27 724.14 bimodal
PerfLabTests.CastingPerf.CheckIsInstAnyIsInterfaceYes 1.13 238137.71 210909.11 bimodal
System.Collections.CopyTo<String>.ReadOnlyMemory(Size: 2048) 1.13 655.73 582.35
System.Collections.CtorDefaultSize<String>.Queue 1.13 15.35 13.64 several?
Benchstone.BenchF.NewtE.Test 1.12 457684900.00 407243450.00
System.Numerics.Tests.Perf_Vector4.GetHashCodeBenchmark 1.12 9.45 8.42
System.Collections.AddGivenSize<Int32>.Stack(Size: 512) 1.12 1152.94 1028.70

@adamsitnik
Copy link
Member Author

adamsitnik commented Dec 14, 2018 •
edited
Loading

x86 Windows 10 PC

Windows 10.0.17763.134 (1809/October2018Update/Redstone5)
Intel Xeon CPU E5-1650 v4 3.60GHz
BenchmarkDotNet 0.11.3.886-nightly
base: .NET Core 2.1.6 (CoreCLR 4.6.27019.06, CoreFX 4.6.27019.05)
diff: .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04)

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.Numerics.Tests.Constructor.ConstructorBenchmark_Int16 9.81 3.24 31.78
System.Threading.Tests.Perf_Interlocked.CompareExchange_long 2.32 8.90 20.64
System.Tests.Perf_String.Equality(s1: "dzsdzsDDZSDZSDZSddsz", s2: "dzsdzsDDZSDZS 2.11 1.51 3.18 bimodal
LinqBenchmarks.Where01LinqMethodX 1.63 417728400.00 679522400.00
System.Memory.Span<Byte>.EndsWith(Size: 512) 1.60 11.06 17.70 several?
System.Memory.Span<Byte>.Clear(Size: 512) 1.55 20.57 31.93 bimodal
System.Memory.Span<Byte>.SequenceEqual(Size: 512) 1.49 14.26 21.25
System.Numerics.Tests.Perf_Vector3.DistanceBenchmark 1.40 2.60 3.64
System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch_Multithreading(tot 1.37 36672150.00 50326875.00 several?
Functions.MathTests.SinDoubleBenchmark 1.35 65368.99 88048.83
LinqBenchmarks.Order00ManualX 1.33 204504100.00 271168250.00 bimodal
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 1000) 1.28 23.96 30.63 several?
System.Collections.CtorDefaultSize<String>.Stack 1.28 13.63 17.38
Benchstone.BenchF.NewtE.Test 1.27 630122900.00 801165800.00
System.Memory.Span<Byte>.SequenceCompareTo(Size: 512) 1.24 48.05 59.49 several?
System.Collections.CtorDefaultSize<String>.SortedList 1.21 28.34 34.31 bimodal
System.Collections.CtorDefaultSize<String>.Queue 1.19 14.00 16.68
System.Globalization.Tests.Perf_CompareInfo.LastIndexOf(culture: es-ES, source: 1.16 18.12 21.08
System.Collections.CtorDefaultSize<String>.SortedDictionary 1.16 71.25 82.80
System.Perf_Convert.ChangeType 1.16 39.46 45.62
System.Memory.Span<Byte>.Fill(Size: 512) 1.15 14.91 17.20
System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "ascii") 1.14 344.86 394.11
Functions.MathTests.AtanSingleBenchmark 1.14 61848.71 70460.66
System.Tests.Perf_Array.ArrayAssign3D 1.13 1248295.96 1415335.63 bimodal
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Numerics.Tests.Constructor.ConstructorBenchmark_Single 6.75 25.48 3.78
SIMD.ConsoleMandel.VectorFloatSinglethreadRawNoInt 4.21 681793550.00 161900300.00
SeekUnroll.Test(boxedIndex: 11) 3.51 16673950100.00 4754797400.00
Burgers.Test3 2.27 1568539550.00 690771900.00
System.Collections.Concurrent.Count<String>.Dictionary(Size: 512) 2.03 13864.18 6836.98
System.Memory.Span<Char>.StartsWith(Size: 512) 1.82 22.72 12.45 several?
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: 1.63 532100.00 327400.00 bimodal
System.Collections.AddDefaultSize<String>.ConcurrentDictionary(Count: 512) 1.47 135629.32 92239.22
System.IO.Compression.Brotli.Compress(file: TestDocument.pdf, level: Optimal) 1.42 19217.54 13578.71 several?
System.Memory.Span<Char>.BinarySearch(Size: 512) 1.40 16.16 11.57 several?
PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 1000) 1.37 29110433.33 21294954.55
PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 100) 1.34 3004502.56 2233945.09
System.Collections.IterateForEach<String>.ConcurrentDictionary(Size: 512) 1.32 26707.91 20269.26 bimodal
System.Numerics.Tests.Constructor.ConstructorBenchmark_Byte 1.29 5.45 4.23
System.Collections.CtorGivenSize<Int32>.ConcurrentDictionary(Size: 512) 1.28 185.81 145.49
System.Memory.Span<Int32>.EndsWith(Size: 512) 1.26 30.45 24.23 several?
System.Collections.IterateForEach<String>.Queue(Size: 512) 1.22 3974.88 3261.11
Span.IndexerBench.Indexer4(length: 1024) 1.21 4835.18 3982.83
SciMark2.kernel.benchSparseMult 1.21 1087211900.00 898676700.00
System.IO.Tests.Perf_StreamWriter.WritePartialCharArray(writeLength: 100) 1.18 1303690350.00 1106478500.00
System.IO.Tests.Perf_FileStream.WriteAsync(BufferSize: 512, TotalSize: 200000) 1.17 7770725.00 6661887.50 several?
System.Linq.Tests.Perf_Linq.Contains_ElementNotFound(size: 100, iterationCount: 1.17 467.48 400.86 bimodal
System.Collections.IterateForEach<Int32>.IEnumerable(Size: 512) 1.14 2095.49 1839.03
System.Tests.Perf_String.PadLeft(n: 18) 1.13 32.97 29.13 bimodal
System.Tests.Perf_Environment.GetLogicalDrives 1.13 425.72 377.97
System.Collections.CtorFromCollection<Int32>.ConcurrentDictionary(Size: 512) 1.12 74530.27 66410.89
System.Collections.Clear<Int32>.Span(Size: 512) 1.12 116.59 104.18
System.Globalization.Tests.Perf_CompareInfo.IsSortable(text: "$") 1.08 33.33 30.75 bimodal

@adamsitnik
Copy link
Member Author

adamsitnik commented Dec 14, 2018 •
edited
Loading

Ubuntu 18.04 x64 PC

ubuntu 18.04
Intel Xeon CPU E5-1650 v4 3.60GHz
BenchmarkDotNet 0.11.3.886-nightly
base: .NET Core 2.1.6 (CoreCLR 4.6.27019.06, CoreFX 4.6.27019.05)
diff: .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04)

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.Collections.CopyTo<Int32>.ImmutableArray(Size: 2048) 3.48 87.45 304.64
System.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 512) 2.00 7233.77 14473.46
Functions.MathTests.CosDoubleBenchmark 1.52 60352.71 92024.60
System.Collections.CtorDefaultSize<String>.ImmutableArray 1.39 6.53 9.09
System.Collections.IndexerSet<Int32>.Dictionary(Size: 512) 1.37 4700.67 6440.06
System.Tests.Perf_String.LastIndexOf(options: CurrentCultureIgnoreCase) 1.34 559206.77 749641.69
System.Collections.CopyTo<Int32>.Array(Size: 2048) 1.29 167.15 215.02
SeekUnroll.Test(boxedIndex: 3) 1.29 1768306956.00 2274037732.50
System.Collections.CopyTo<Int32>.ReadOnlySpan(Size: 2048) 1.27 171.11 217.70
System.Collections.CopyTo<Int32>.Span(Size: 2048) 1.26 172.62 217.96
System.Collections.ContainsFalse<Int32>.Stack(Size: 512) 1.26 108653.72 136997.03
PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo 1.26 680309.87 855663.61
PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo 1.25 682509.91 855778.84
PerfLabTests.LowLevelPerf.TypeReflectionArrayGetType 1.25 201338.13 251762.33
System.Collections.CopyTo<Int32>.ReadOnlyMemory(Size: 2048) 1.23 187.92 232.04
CscBench.CompileTest 1.23 436566092.00 538142593.00 several?
PerfLabTests.CastingPerf.ObjFooIsObj2 1.23 205002.43 251748.75
System.Collections.ContainsTrue<Int32>.Stack(Size: 512) 1.23 59170.73 72596.56
PerfLabTests.CastingPerf.FooObjIsNull 1.23 205230.22 251721.50
PerfLabTests.CastingPerf.ObjScalarValueType 1.22 205579.12 251758.72
PerfLabTests.CastingPerf.ObjFooIsObj 1.22 205548.32 251715.12
PerfLabTests.CastingPerf.IFooFooIsIFoo 1.22 205651.69 251741.76
PerfLabTests.CastingPerf.ObjObjrefValueType 1.22 205694.57 251738.97
PerfLabTests.CastingPerf.ObjInt 1.22 205869.19 251748.03
System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit 1.21 2277.36 2760.67
System.Globalization.Tests.Perf_CompareInfo.IsSuffix(culture: ja-JP, source: "AB 1.20 3815.34 4586.28
System.Collections.CopyTo<Int32>.Memory(Size: 2048) 1.20 197.20 236.86
System.Collections.IterateForEach<Int32>.ImmutableStack(Size: 512) 1.20 3547.73 4260.24 bimodal
MicroBenchmarks.Serializers.Json_ToString<LoginViewModel>.Jil_ 1.17 2868.50 3350.00 several?
PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 1000) 1.16 30355751.56 35335613.36
System.Collections.IterateFor<Int32>.ImmutableSortedSet(Size: 512) 1.16 8965.52 10393.59
System.Tests.Perf_String.IndexerCheckPathLength 1.16 112.89 130.76
System.Tests.Perf_Array.ArrayRetrieve1D 1.15 2983349.88 3432347.94
System.Collections.Concurrent.AddRemoveFromSameThreads<String>.ConcurrentBag(Siz 1.15 306800366.20 352483392.65
System.Tests.Perf_Environment.GetFolderPath(folder: System, option: None) 1.15 615.26 704.61
System.Collections.Concurrent.AddRemoveFromSameThreads<String>.ConcurrentStack(S 1.14 227790276.40 258564017.95
System.Globalization.Tests.Perf_CompareInfo.IndexOf(culture: , source: "More Tes 1.13 4375.81 4944.37
PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo 1.13 403783.59 455042.52
MicroBenchmarks.Serializers.Binary_FromStream<LoginViewModel>.BinaryFormatter_ 1.13 3230.10 3637.15
PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj 1.12 403254.42 453563.90
PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull 1.12 403893.38 453546.82
MicroBenchmarks.Serializers.Binary_FromStream<CollectionsOfPrimitives>.BinaryFor 1.12 1596551.94 1790917.54
System.Collections.CtorDefaultSizeNonGeneric.SortedList 1.12 40.60 45.53
System.Memory.Constructors<String>.MemoryFromArray 1.11 12.80 14.25
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Numerics.Tests.Constructor.SpanCastBenchmark_SByte 16.20 8.84 0.55
System.Collections.Concurrent.Count<String>.Dictionary(Size: 512) 3.98 15303.11 3848.54
System.Memory.MemoryMarshal<Byte>.Read 3.79 7.59 2.00
Benchstone.BenchF.Trap.Test 1.57 371708036.00 236563752.50
LinqBenchmarks.Where01LinqQueryX 1.52 846559723.00 556294327.00
System.Collections.CtorDefaultSize<String>.ImmutableSortedSet 1.47 8.17 5.57
System.Tests.Perf_Array.ArrayCopy3D 1.36 329.56 243.05
System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark 1.31 5.12 3.90
System.IO.Tests.Perf_FileStream.ReadAsync(BufferSize: 200000, TotalSize: 200000) 1.31 42684.09 32646.96 several?
SeekUnroll.Test(boxedIndex: 1) 1.28 2273664575.00 1770272642.50
Functions.MathTests.TanhDoubleBenchmark 1.23 82115.73 66559.42
System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja) 1.21 5257.00 4333.50 several?
MicroBenchmarks.Serializers.Xml_ToStream<XmlElement>.DataContractSerializer_ 1.21 597.47 492.63
System.Memory.Span<Char>.LastIndexOfValue(Size: 512) 1.20 23.15 19.24
MicroBenchmarks.Serializers.Binary_FromStream<CollectionsOfPrimitives>.MessagePa 1.20 103737.11 86319.97
System.Collections.Clear<String>.Array(Size: 512) 1.17 217.56 185.24
System.Tests.Perf_String.ToLower(s: "TEST") 1.17 40.97 35.00
System.Memory.Span<Byte>.LastIndexOfValue(Size: 512) 1.17 17.77 15.22
System.Tests.Perf_String.Concat_str_str(size: 100) 1.16 49.04 42.13
System.Globalization.Tests.Perf_CompareInfo.LastIndexOf(culture: , source: "?", 1.16 4858.72 4192.81
System.Collections.CtorDefaultSize<String>.HashSet 1.15 20.28 17.57
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 1000) 1.15 18.43 16.00
PerfLabTests.CastingPerf2.CastingPerf.FooObjIsDescendant 1.15 608606.18 528896.74
System.Collections.IterateForEach<String>.HashSet(Size: 512) 1.15 2767.20 2406.88
System.Memory.Span<Char>.Clear(Size: 512) 1.14 11.11 9.74
System.Collections.IterateForEach<Int32>.ImmutableQueue(Size: 512) 1.14 4151.92 3642.74
MicroBenchmarks.Serializers.Xml_ToStream<XmlElement>.XmlSerializer_ 1.14 1731.26 1520.11
Benchstone.BenchI.Ackermann.Test 1.14 3982.75 3502.19
System.Memory.Span<Int32>.Clear(Size: 512) 1.14 21.10 18.56
System.Memory.Span<Byte>.ToArray(Size: 512) 1.13 51.04 45.23
System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", 1.13 80.54 71.50
PerfLabTests.LowLevelPerf.TypeReflectionObjectGetType 1.13 226536.28 201337.69
System.Tests.Perf_String.ToUpper(s: "TEST") 1.12 30.35 27.05
System.Tests.Perf_Int64.Parse(value: "12345") 1.12 85.60 76.32
System.Collections.CtorFromCollection<String>.ConcurrentDictionary(Size: 512) 1.12 158699.77 141632.99
System.Text.Tests.Perf_Encoding.GetEncoder(size: 512, encName: "ascii") 1.12 10.87 9.71
System.Text.Tests.Perf_Encoding.GetEncoder(size: 512, encName: "utf-8") 1.12 10.93 9.77
System.Collections.TryAddGiventSize<String>.ConcurrentDictionary(Count: 512) 1.12 93533.00 83600.69
XmlDocumentTests.XmlNodeListTests.Perf_XmlNodeList.GetCount 1.11 16.00 14.37
System.IO.Tests.Perf_Path.GetFileName 1.11 68.62 61.72

@adamsitnik
Copy link
Member Author

adamsitnik commented Dec 14, 2018 •
edited
Loading

macOS 10.13.6 x64 MacBook Pro

macOS High Sierra 10.13.6 (17G65) [Darwin 17.7.0]
Intel Core i7-5557U CPU 3.10GHz (Broadwell)
BenchmarkDotNet 0.11.3.886-nightly
base: .NET Core 2.1.5 (CoreCLR 4.6.26919.02, CoreFX 4.6.26919.02)
diff: .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04)

Slower diff/base Base Median (ns) Diff Median (ns) Modality
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 100) 1.60 9.22 14.76
System.Tests.Perf_String.Trim_CharArr(s: "Test", c: [' ', ' ']) 1.41 6.18 8.72
System.Collections.AddGivenSize<String>.HashSet(Size: 512) 1.34 26884.73 36023.49
System.Tests.Perf_String.TrimStart_CharArr(s: "Test", c: [' ', ' ']) 1.31 5.26 6.86
System.Collections.Remove<Int32>.LinkedList(Size: 512) 1.26 6627.86 8361.80
System.Collections.ContainsTrue<String>.Array(Size: 512) 1.25 457193.13 569259.01
System.Collections.AddDefaultSize<String>.HashSet(Count: 512) 1.23 34303.63 42303.68
System.Tests.Perf_String.PadLeft(n: 18) 1.23 19.24 23.60
System.Collections.ContainsFalse<String>.Array(Size: 512) 1.22 903160.89 1105794.22
System.Tests.Perf_String.TrimEnd_CharArr(s: "Test", c: [' ', ' ']) 1.21 5.49 6.63
System.Collections.IterateForEach<Int32>.LinkedList(Size: 512) 1.21 2147.11 2593.48
System.Collections.IterateForEach<Int32>.ImmutableStack(Size: 512) 1.19 4148.35 4916.46
System.Tests.Perf_String.IndexerCheckPathLength 1.16 131.95 152.62
System.Collections.IterateFor<Int32>.ImmutableSortedSet(Size: 512) 1.15 10538.73 12150.65
System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], opti 1.15 53.04 60.81
System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' ']) 1.14 19.79 22.58
System.Collections.Remove<Int32>.ConcurrentBag(Size: 512) 1.14 14150.20 16130.23
MicroBenchmarks.Serializers.Json_ToString<LoginViewModel>.JsonNet_ 1.13 1024.52 1162.35
PerfLabTests.LowLevelPerf.TypeReflectionArrayGetType 1.13 235416.45 264849.59
System.Tests.Perf_Int32.Parse(value: "-2147483648") 1.12 110.69 124.16
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Collections.Clear<String>.List(Size: 512) 1.50 475.61 317.54 several?
System.Memory.Span<Char>.EndsWith(Size: 512) 1.40 23.89 17.03
System.Collections.IterateForEach<String>.ConcurrentDictionary(Size: 512) 1.31 32192.90 24573.18
System.Tests.Perf_Array.ArrayCopy3D 1.31 372.71 284.73
System.Tests.Perf_String.TrimEnd(s: "Test") 1.29 6.19 4.79
System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 1.24 176815.16 142338.71
System.Memory.Span<Char>.LastIndexOfValue(Size: 512) 1.21 27.22 22.50
PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo 1.20 1030250.82 859290.01
SeekUnroll.Test(boxedIndex: 3) 1.20 2657032251.00 2220589998.00
PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo 1.20 1029828.35 860882.13
SeekUnroll.Test(boxedIndex: 1) 1.19 2657148998.50 2223774490.00
SeekUnroll.Test(boxedIndex: 11) 1.19 2453818595.00 2064674343.00
System.Collections.ContainsFalse<Int32>.HashSet(Size: 512) 1.18 7320.35 6179.89
Functions.MathTests.SinhSingleBenchmark 1.18 32890.05 27855.95
System.Collections.ContainsKeyFalse<Int32, Int32>.Dictionary(Size: 512) 1.18 4388.64 3722.53
System.Memory.Span<Byte>.LastIndexOfValue(Size: 512) 1.18 20.81 17.68
System.Threading.Tasks.ValueTaskPerfTest.Copy_PassAsArgumentAndReturn_FromResult 1.17 25.39 21.66
System.Threading.Tasks.ValueTaskPerfTest.Copy_PassAsArgumentAndReturn_FromTask 1.16 27.15 23.44
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10) 1.16 7.68 6.65
System.Collections.Remove<String>.LinkedList(Size: 512) 1.15 15836.06 13726.43
Benchstone.BenchI.Permutate.Test 1.15 862684807.50 747822615.00
System.Perf_Convert.GetTypeCode 1.15 10.31 8.95
System.Globalization.Tests.Perf_CultureInfo.GetCurrentCulture 1.14 9.03 7.89
Benchstone.BenchI.Ackermann.Test 1.14 4687.92 4104.95
System.Collections.AddGivenSize<Int32>.Dictionary(Size: 512) 1.13 9049.80 7990.51
System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 512, TotalSize: 200000) 1.13 1126631.26 998725.58
PerfLabTests.LowLevelPerf.TypeReflectionObjectGetType 1.12 264830.39 235413.78
Benchstone.BenchF.Simpsn.Test 1.12 166082870.00 148617704.50
System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 200000, TotalSize: 200000) 1.12 1118211.45 1001192.44
System.Collections.AddDefaultSize<String>.SortedList(Count: 512) 1.11 764247.23 686386.12

@adamsitnik
Copy link
Member Author

System.Collections.ContainsKeyFalse<Int32, Int32>.Dictionary(Size: 512) (eg) appears in both faster and slower. Can you clarify ?

@danmosemsft this benchmark is not stable. I guess that it's dependent on array or loop alignment. I need to to a research about the alignment issues.

@danmoseley
Copy link
Member

@adamsitnik I guess most of these must be either noise, alignment or tiered jitting, given there were of course very few changes made in 2.2 corefx/coreclr that were not serviced into 2.1. Would it be interesting to repeat the 2.1->2.2 analysis with tiered jitting disabled? Then if the system is working well, I would expect just about zero regressions/improvements.

The 2.2->3.0 report will be much more interesting...

@danmoseley
Copy link
Member

danmoseley commented Dec 18, 2018 •
edited
Loading

Another thing that might be interesting is to see what it looks like with the OS aggregated. I did a crude comparison of the regressions above by sort and diff and found only a very few of the rows appear in all the OS. In general most (?) times we regress something, it regresses for all OS. So it could be useful interesting pointer to particularly interesting regressions.

@adamsitnik
Copy link
Member Author

I am closing this issue since we did not find any significant regressions. There was some noise, but having 20-30 noisy benchmarks in a portfolio of 2000 benchmarks is quite a good result ;)

@danmosemsft let's talk about the other comparisons we need during our next 1:1

@danmoseley
Copy link
Member

OK!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adamsitnik @danmoseley