Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 221 Improvements on 5/29/2023 1:48:21 PM #18537

Closed
performanceautofiler bot opened this issue Jun 6, 2023 · 12 comments
Closed

[Perf] Linux/x64: 221 Improvements on 5/29/2023 1:48:21 PM #18537

performanceautofiler bot opened this issue Jun 6, 2023 · 12 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 126.64 ns 6.27 ns 0.05 0.13 True
InequalityOperatorBenchmark - Duration of single invocation 18.22 ns 9.41 ns 0.52 0.35 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.38 ns 8.11 ns 0.21 0.19 True
AddBenchmark - Duration of single invocation 123.46 ns 8.21 ns 0.07 0.12 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.58 ns 10.28 ns 0.27 0.23 True
MultiplyOperatorBenchmark - Duration of single invocation 123.72 ns 8.28 ns 0.07 0.14 True
AddOperatorBenchmark - Duration of single invocation 123.62 ns 8.21 ns 0.07 0.14 True
BitwiseOrOperatorBenchmark - Duration of single invocation 39.72 ns 8.16 ns 0.21 0.20 True
SubtractBenchmark - Duration of single invocation 125.44 ns 8.32 ns 0.07 0.12 True
EqualsAllBenchmark - Duration of single invocation 17.85 ns 7.13 ns 0.40 0.26 True
OnesComplementBenchmark - Duration of single invocation 29.00 ns 6.14 ns 0.21 0.23 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.11 ns 6.26 ns 0.22 0.21 True
AndNotBenchmark - Duration of single invocation 65.47 ns 8.36 ns 0.13 0.20 True
XorBenchmark - Duration of single invocation 38.38 ns 8.21 ns 0.21 0.22 True
OneBenchmark - Duration of single invocation 51.26 ns 1.43 ns 0.03 0.20 True
MultiplyBenchmark - Duration of single invocation 123.86 ns 8.27 ns 0.07 0.12 True
EqualityOperatorBenchmark - Duration of single invocation 18.05 ns 7.28 ns 0.40 0.28 True
NegateBenchmark - Duration of single invocation 125.76 ns 6.31 ns 0.05 0.11 True
BitwiseAndBenchmark - Duration of single invocation 38.17 ns 8.19 ns 0.21 0.20 True
SubtractionOperatorBenchmark - Duration of single invocation 125.79 ns 8.18 ns 0.07 0.16 True
BitwiseOrBenchmark - Duration of single invocation 38.38 ns 7.93 ns 0.21 0.23 True
ConditionalSelectBenchmark - Duration of single invocation 162.26 ns 15.09 ns 0.09 0.19 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.265441118834393 < 119.23574007237914.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 258.1893226567115 (T) = (0 -6.736359415948462) / Math.Sqrt((28.214044610745894 / (299)) + (3.6255173251145507 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9493734572732528 = (133.05983488359905 - 6.736359415948462) / 133.05983488359905 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.408263579882517 < 17.22307257079854.
IsChangePoint: Marked as a change because one of 4/17/2023 1:30:24 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 46.119327031267325 (T) = (0 -7.603655565737102) / Math.Sqrt((2.827904341573698 / (299)) + (1.3355885373894318 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6033278597089637 = (19.1686150687526 - 7.603655565737102) / 19.1686150687526 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.111328988904988 < 36.467597047965896.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 294.8571266559773 (T) = (0 -8.121970427951762) / Math.Sqrt((3.0086088320983992 / (299)) + (0.03528506364695466 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.795445287551123 = (39.70561387082016 - 8.121970427951762) / 39.70561387082016 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.214735147499955 < 117.99644279038408.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 329.53265515113105 (T) = (0 -8.392623135562728) / Math.Sqrt((30.1579351551795 / (299)) + (1.0709569667804117 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9370450431853581 = (133.3115541683731 - 8.392623135562728) / 133.3115541683731 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.27731041264066 < 36.51342493148057.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 115.55964724725426 (T) = (0 -8.787659202527006) / Math.Sqrt((6.865618718591095 / (299)) + (1.257275608094183 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.780660592189037 = (40.064205927375475 - 8.787659202527006) / 40.064205927375475 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.277644316271976 < 117.73021441749472.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 277.1984160518003 (T) = (0 -8.17616498071617) / Math.Sqrt((56.97748619450631 / (299)) + (0.3239168177797835 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9386303915929739 = (133.22824102915564 - 8.17616498071617) / 133.22824102915564 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.208412794640868 < 117.50027474495924.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 290.19047083078016 (T) = (0 -8.220808960029238) / Math.Sqrt((49.32198793238665 / (299)) + (0.539795830590042 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9384474961643915 = (133.55766943266818 - 8.220808960029238) / 133.55766943266818 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.15899047261179 < 36.36369295743081.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 217.41620614101345 (T) = (0 -8.240106610606263) / Math.Sqrt((4.88795284677043 / (299)) + (0.12371595533610567 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.793833463690721 = (39.968206082896685 - 8.240106610606263) / 39.968206082896685 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.321805471048297 < 119.10032743798274.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 412.9097242418427 (T) = (0 -8.244559326403852) / Math.Sqrt((22.860774772717985 / (299)) + (0.386232111524605 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.938207256388658 = (133.42277498244252 - 8.244559326403852) / 133.42277498244252 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.133679158304825 < 17.191293747409.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 61.248600766403406 (T) = (0 -7.370084156959092) / Math.Sqrt((2.7590235519607527 / (299)) + (0.6926441766581514 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.614955414270992 = (19.140859085202436 - 7.370084156959092) / 19.140859085202436 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.142362473584433 < 27.643798077685986.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 119.24385572904049 (T) = (0 -6.46702244887348) / Math.Sqrt((3.494236061146733 / (299)) + (0.712857201993146 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7870977538321605 = (30.37554823999021 - 6.46702244887348) / 30.37554823999021 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.263621035216482 < 27.66671196629385.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 193.24947001665157 (T) = (0 -6.412030692575108) / Math.Sqrt((2.907450641463267 / (299)) + (0.1385266066326701 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.788300062450522 = (30.288297515801116 - 6.412030692575108) / 30.288297515801116 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.359603620667134 < 62.44939236273636.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 216.60610121257798 (T) = (0 -9.612203942427485) / Math.Sqrt((11.156132410753772 / (299)) + (0.8075965549220151 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8560252348529085 = (66.76311597110227 - 9.612203942427485) / 66.76311597110227 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.211196483284466 < 36.73613557533039.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 190.5948167368194 (T) = (0 -8.202849000629511) / Math.Sqrt((4.2369510221369735 / (299)) + (0.33245683025669553 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7938538200668769 = (39.791418901338055 - 8.202849000629511) / 39.791418901338055 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.433502432247066 < 48.63710596463662.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 193.43381109387255 (T) = (0 -1.7684388920596157) / Math.Sqrt((21.493797921617148 / (299)) + (0.17318887823540738 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9684615277066958 = (56.07243355395704 - 1.7684388920596157) / 56.07243355395704 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.265849407162026 < 117.68749181352081.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 332.7935175499894 (T) = (0 -8.094729039048119) / Math.Sqrt((41.71972102817896 / (299)) + (0.03547767801008012 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9391540290451991 = (133.036401786755 - 8.094729039048119) / 133.036401786755 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.2809534417977035 < 17.042486613174308.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 92.64058794070928 (T) = (0 -7.190589461203351) / Math.Sqrt((2.4857260651540374 / (299)) + (0.21189515594406952 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6253798403778521 = (19.19434733159042 - 7.190589461203351) / 19.19434733159042 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.313720300658882 < 119.33392829757082.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 371.00795868370494 (T) = (0 -6.591065308121428) / Math.Sqrt((27.62276989610305 / (299)) + (0.5945655214623492 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9504589330354392 = (133.0424577419042 - 6.591065308121428) / 133.0424577419042 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.189594803720516 < 36.77727571601067.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 191.31162607824282 (T) = (0 -8.10208945290391) / Math.Sqrt((6.68103071069672 / (299)) + (0.13541144940204147 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7973350512634859 = (39.977753940260705 - 8.10208945290391) / 39.977753940260705 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.181660944221967 < 119.33140420049044.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 415.1211386357434 (T) = (0 -8.145684372326881) / Math.Sqrt((25.431844885980787 / (299)) + (0.16151964778498049 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9390873440012147 = (133.72728932537962 - 8.145684372326881) / 133.72728932537962 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.9336385090861095 < 36.66498791926264.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 186.00935238167108 (T) = (0 -8.115661164652465) / Math.Sqrt((7.373767906070478 / (299)) + (0.12973404545547534 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7983847336564124 = (40.253207566246296 - 8.115661164652465) / 40.253207566246296 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int16>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.091095640109588 < 156.08426576587726.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 354.4232236115261 (T) = (0 -14.582600880376063) / Math.Sqrt((44.36163878912075 / (299)) + (0.4886517616249302 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9087533366351562 = (159.81516849628213 - 14.582600880376063) / 159.81516849628213 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.39 ns 8.38 ns 0.22 0.20 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.53 ns 8.76 ns 0.23 0.22 True
OnesComplementOperatorBenchmark - Duration of single invocation 34.94 ns 6.42 ns 0.18 0.17 True
BitwiseOrBenchmark - Duration of single invocation 38.34 ns 8.22 ns 0.21 0.18 True
InequalityOperatorBenchmark - Duration of single invocation 18.05 ns 7.45 ns 0.41 0.33 True
ConditionalSelectBenchmark - Duration of single invocation 161.58 ns 14.91 ns 0.09 0.19 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.28 ns 10.55 ns 0.28 0.23 True
EqualsAllBenchmark - Duration of single invocation 18.09 ns 6.97 ns 0.39 0.28 True
EqualityOperatorBenchmark - Duration of single invocation 18.21 ns 6.88 ns 0.38 0.29 True
OneBenchmark - Duration of single invocation 19.04 ns 1.48 ns 0.08 0.27 True
BitwiseAndBenchmark - Duration of single invocation 38.92 ns 8.24 ns 0.21 0.18 True
AndNotBenchmark - Duration of single invocation 65.14 ns 9.61 ns 0.15 0.27 True
OnesComplementBenchmark - Duration of single invocation 29.10 ns 8.94 ns 0.31 0.24 True
XorBenchmark - Duration of single invocation 36.45 ns 8.14 ns 0.22 0.18 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.37601168594269 < 36.566403674306315.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 205.4722257320158 (T) = (0 -8.181889358202866) / Math.Sqrt((6.004252421630087 / (299)) + (0.09431054558650061 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7950234969401077 = (39.916230573082764 - 8.181889358202866) / 39.916230573082764 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.755748981818833 < 36.46519752034313.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 184.11199669194528 (T) = (0 -8.285814645639883) / Math.Sqrt((4.946129181865271 / (299)) + (0.3312130009382985 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7931831844142597 = (40.063544263424866 - 8.285814645639883) / 40.063544263424866 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.4150786825401385 < 27.72111520237693.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 161.72304853070952 (T) = (0 -6.603760667334717) / Math.Sqrt((1.7343606557836766 / (299)) + (0.3837912248449287 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.780783830330812 = (30.124423199713043 - 6.603760667334717) / 30.124423199713043 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.22347040848368 < 36.53707605207939.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 247.5763291709476 (T) = (0 -8.187408113890182) / Math.Sqrt((3.4353029447024817 / (299)) + (0.12361681389984203 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7949327410217737 = (39.92547691272115 - 8.187408113890182) / 39.92547691272115 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.4520332869779455 < 17.31492790124936.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 63.37506869469482 (T) = (0 -7.324844341953229) / Math.Sqrt((2.7640893420292794 / (299)) + (0.6697865273387197 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6215609466797383 = (19.355413448184564 - 7.324844341953229) / 19.355413448184564 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.908818862852286 < 154.46554291552036.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 185.53827862808353 (T) = (0 -15.64083966397555) / Math.Sqrt((40.55732660051286 / (299)) + (11.622359910214895 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9018906673935787 = (159.42254674915458 - 15.64083966397555) / 159.42254674915458 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.550462435719279 < 36.38251683637981.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 170.72486085153508 (T) = (0 -8.22411377302114) / Math.Sqrt((6.635454144605183 / (299)) + (0.3072764225766566 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7940219285372109 = (39.92713260502034 - 8.22411377302114) / 39.92713260502034 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.9668604568131824 < 17.215731282057337.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 84.79196821110054 (T) = (0 -7.210251582006095) / Math.Sqrt((2.732012213044902 / (299)) + (0.2794592414694771 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6263318536238152 = (19.29586894663288 - 7.210251582006095) / 19.29586894663288 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.883307274361262 < 17.2926403321435.
IsChangePoint: Marked as a change because one of 4/12/2023 4:47:47 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 61.887315128665534 (T) = (0 -7.402497141741716) / Math.Sqrt((2.742413560478385 / (299)) + (0.6807606715414231 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6146585907367508 = (19.210230107101307 - 7.402497141741716) / 19.210230107101307 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4758325789488806 < 18.390223988912584.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 244.4140021675673 (T) = (0 -1.6736377876430597) / Math.Sqrt((1.000131218899909 / (299)) + (0.05803511623783206 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9166178734124294 = (20.071900971311283 - 1.6736377876430597) / 20.071900971311283 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.235417850646817 < 36.429306852505576.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 218.39039023342048 (T) = (0 -8.14341920607535) / Math.Sqrt((3.9562087596177133 / (299)) + (0.18621555135657705 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.7953091666052254 = (39.783995555724964 - 8.14341920607535) / 39.783995555724964 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.61151259985576 < 62.13186629024071.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 243.47972922884742 (T) = (0 -9.439219656912735) / Math.Sqrt((12.734046304681602 / (299)) + (0.3275623964633048 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8589013189263062 = (66.8980006410036 - 9.439219656912735) / 66.8980006410036 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.939021851341744 < 27.628137198595613.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 135.17780769141032 (T) = (0 -6.479609484986395) / Math.Sqrt((3.1802163816753404 / (299)) + (0.5149320313401841 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7866413784393675 = (30.369569495671925 - 6.479609484986395) / 30.369569495671925 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.137227214254724 < 36.40947953683676.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 175.32752145169118 (T) = (0 -8.30297604801724) / Math.Sqrt((3.969167415878273 / (299)) + (0.46888460115229763 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7907585119553571 = (39.68130854740312 - 8.30297604801724) / 39.68130854740312 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualityOperatorBenchmark - Duration of single invocation 23.26 ns 7.97 ns 0.34 0.29 True
OneBenchmark - Duration of single invocation 76.32 ns 1.53 ns 0.02 0.15 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.30 ns 8.06 ns 0.21 0.16 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.47 ns 8.19 ns 0.21 0.20 True
SubtractionOperatorBenchmark - Duration of single invocation 206.99 ns 8.19 ns 0.04 0.13 True
InequalityOperatorBenchmark - Duration of single invocation 18.51 ns 8.01 ns 0.43 0.28 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.14 ns 6.02 ns 0.21 0.22 True
AddOperatorBenchmark - Duration of single invocation 200.23 ns 11.42 ns 0.06 0.09 True
ConditionalSelectBenchmark - Duration of single invocation 166.86 ns 15.10 ns 0.09 0.21 True
SubtractBenchmark - Duration of single invocation 203.26 ns 8.05 ns 0.04 0.07 True
AndNotBenchmark - Duration of single invocation 65.40 ns 9.57 ns 0.15 0.27 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.35 ns 8.08 ns 0.21 0.24 True
OnesComplementBenchmark - Duration of single invocation 29.10 ns 5.89 ns 0.20 0.19 True
XorBenchmark - Duration of single invocation 41.96 ns 8.39 ns 0.20 0.14 True
MultiplyBenchmark - Duration of single invocation 201.01 ns 8.43 ns 0.04 0.10 True
EqualsAllBenchmark - Duration of single invocation 19.96 ns 7.08 ns 0.35 0.26 True
AddBenchmark - Duration of single invocation 200.29 ns 8.23 ns 0.04 0.07 True
BitwiseAndBenchmark - Duration of single invocation 38.32 ns 8.20 ns 0.21 0.22 True
BitwiseOrBenchmark - Duration of single invocation 38.04 ns 8.13 ns 0.21 0.23 True
NegateBenchmark - Duration of single invocation 206.56 ns 6.26 ns 0.03 0.11 True
MultiplyOperatorBenchmark - Duration of single invocation 225.02 ns 8.20 ns 0.04 0.11 True
UnaryNegateOperatorBenchmark - Duration of single invocation 210.80 ns 6.29 ns 0.03 0.09 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.969284710706546 < 17.716031891212694.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 38.478774129645345 (T) = (0 -8.05319415206578) / Math.Sqrt((3.182321373859595 / (299)) + (1.9277385041966653 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.5859922712181735 = (19.451796650660228 - 8.05319415206578) / 19.451796650660228 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.5307616615664472 < 72.45824855815003.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 251.7392901539321 (T) = (0 -1.6652594532465246) / Math.Sqrt((31.281993036554415 / (299)) + (0.09263203061801918 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9802975693406166 = (84.52050825786996 - 1.6652594532465246) / 84.52050825786996 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.058443055807496 < 36.41376336935511.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 231.76184092028336 (T) = (0 -8.22037371462445) / Math.Sqrt((3.41179991955748 / (299)) + (0.17312092552007405 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.792430336404382 = (39.60296303528811 - 8.22037371462445) / 39.60296303528811 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.19492062614346 < 36.51326128918717.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 158.75647221029135 (T) = (0 -8.359598416509096) / Math.Sqrt((4.149195362866191 / (299)) + (0.6287881584578681 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7895522493841698 = (39.72291645810669 - 8.359598416509096) / 39.72291645810669 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.189964536669907 < 196.71188171071148.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 452.55652264209624 (T) = (0 -8.082666338354551) / Math.Sqrt((56.41572275609063 / (299)) + (0.4015292112953505 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9620278279725835 = (212.85762459199663 - 8.082666338354551) / 212.85762459199663 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.014192465359814 < 17.602737795559566.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 76.84314633043101 (T) = (0 -8.222241284666817) / Math.Sqrt((1.0621693750735361 / (299)) + (0.4225833202400136 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.5720395978477423 = (19.212621642834954 - 8.222241284666817) / 19.212621642834954 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.017457193823792 < 27.653258503062744.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 188.00750853996843 (T) = (0 -6.119359217762025) / Math.Sqrt((2.323045328369259 / (299)) + (0.218113422442358 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7978067965234543 = (30.264910553592706 - 6.119359217762025) / 30.264910553592706 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.424587816749936 < 190.38343034722175.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 369.09886265808314 (T) = (0 -8.445624699565512) / Math.Sqrt((79.5150597630828 / (299)) + (0.6423616259850544 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9593510174613619 = (207.7696456864004 - 8.445624699565512) / 207.7696456864004 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.099464326443721 < 158.2239759557624.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 356.5092144053413 (T) = (0 -14.785577725735374) / Math.Sqrt((42.17028585278299 / (299)) + (0.6168652777355298 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9075396175214437 = (159.9125736816468 - 14.785577725735374) / 159.9125736816468 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.046458235732334 < 197.2271976872936.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 559.8315480484926 (T) = (0 -8.404577176995007) / Math.Sqrt((35.05966657647243 / (299)) + (0.3752280936849006 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9603568816102706 = (212.0059550908696 - 8.404577176995007) / 212.0059550908696 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.5659935029722 < 63.06280663720778.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 244.59080711698306 (T) = (0 -9.171814342121104) / Math.Sqrt((14.767443874409349 / (299)) + (0.17224665895636113 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8635080250043697 = (67.19672964227188 - 9.171814342121104) / 67.19672964227188 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.080326959195151 < 36.65728433005096.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 147.3876191021155 (T) = (0 -8.499030224841421) / Math.Sqrt((6.775593004156441 / (299)) + (0.5913998491515547 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7886803320747767 = (40.21883201069979 - 8.499030224841421) / 40.21883201069979 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.892703027029709 < 27.701087221241025.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 194.8272198445614 (T) = (0 -6.384356908546439) / Math.Sqrt((1.616933755063163 / (299)) + (0.23744793896283525 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7883915655369366 = (30.170616425314744 - 6.384356908546439) / 30.170616425314744 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.388334801032338 < 36.81186977445677.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 294.74330274168045 (T) = (0 -8.068703119944667) / Math.Sqrt((2.7297061415358828 / (299)) + (0.057733437338218764 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7962045766659568 = (39.592170363507954 - 8.068703119944667) / 39.592170363507954 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.434904016650822 < 190.61713548458957.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 475.2901961261488 (T) = (0 -8.807602996427981) / Math.Sqrt((44.697564983329826 / (299)) + (0.6247655816525549 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9575210874129813 = (207.34059466295128 - 8.807602996427981) / 207.34059466295128 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.08384642764605 < 17.566568420485996.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 71.59653114088721 (T) = (0 -8.3821498543965) / Math.Sqrt((1.6421529630257037 / (299)) + (0.44049540803778714 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.5649410423761282 = (19.26669870257734 - 8.3821498543965) / 19.26669870257734 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.227530671783956 < 190.4762209197981.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 422.39066837692513 (T) = (0 -8.306642906655178) / Math.Sqrt((58.27924332742553 / (299)) + (0.7118271469640446 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9600535852190974 = (207.94464164594785 - 8.306642906655178) / 207.94464164594785 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.204615198700806 < 36.38399968949204.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 187.9900293718919 (T) = (0 -8.22597195769497) / Math.Sqrt((4.813281206770888 / (299)) + (0.3092196088599657 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7940613763602047 = (39.94380370378173 - 8.22597195769497) / 39.94380370378173 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.131625772498177 < 36.45307926921027.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 176.01852225795318 (T) = (0 -8.300910440740191) / Math.Sqrt((6.087742311835851 / (299)) + (0.3087599575201717 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7931794745524837 = (40.13581545051565 - 8.300910440740191) / 40.13581545051565 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.259308653197357 < 196.42739106483262.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 355.0506230049339 (T) = (0 -6.438651549563909) / Math.Sqrt((97.65873990200102 / (299)) + (0.21657034203700473 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9696326012178473 = (212.02512588427473 - 6.438651549563909) / 212.02512588427473 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.19837378212373 < 190.58903102831246.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 287.76417277585585 (T) = (0 -8.73523638291131) / Math.Sqrt((138.1386451169523 / (299)) + (0.5133646831016504 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9581305337254421 = (208.63023009728036 - 8.73523638291131) / 208.63023009728036 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.287297525218119 < 196.4985215573601.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 393.2007188071887 (T) = (0 -6.472608647550769) / Math.Sqrt((77.69213462729621 / (299)) + (0.34088527919158024 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.969482704899368 = (212.0964071752458 - 6.472608647550769) / 212.0964071752458 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualityOperatorBenchmark - Duration of single invocation 18.24 ns 7.20 ns 0.39 0.31 True
XorBenchmark - Duration of single invocation 38.37 ns 9.76 ns 0.25 0.18 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 45.96 ns 8.05 ns 0.18 0.19 True
BitwiseOrBenchmark - Duration of single invocation 38.39 ns 8.31 ns 0.22 0.22 True
AndNotBenchmark - Duration of single invocation 65.59 ns 9.60 ns 0.15 0.22 True
EqualsAllBenchmark - Duration of single invocation 18.01 ns 7.47 ns 0.41 0.28 True
UnaryNegateOperatorBenchmark - Duration of single invocation 128.63 ns 6.25 ns 0.05 0.16 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.50 ns 8.21 ns 0.21 0.24 True
OnesComplementBenchmark - Duration of single invocation 29.36 ns 6.28 ns 0.21 0.22 True
MultiplyBenchmark - Duration of single invocation 184.66 ns 8.06 ns 0.04 0.14 True
AddOperatorBenchmark - Duration of single invocation 124.68 ns 8.17 ns 0.07 0.12 True
InequalityOperatorBenchmark - Duration of single invocation 18.27 ns 7.10 ns 0.39 0.29 True
OneBenchmark - Duration of single invocation 51.22 ns 1.17 ns 0.02 0.14 True
AddBenchmark - Duration of single invocation 124.15 ns 8.11 ns 0.07 0.16 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.49 ns 8.25 ns 0.21 0.22 True
BitwiseAndBenchmark - Duration of single invocation 38.42 ns 10.34 ns 0.27 0.21 True
ConditionalSelectBenchmark - Duration of single invocation 161.67 ns 15.21 ns 0.09 0.21 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.17 ns 5.57 ns 0.19 0.22 True
MultiplyOperatorBenchmark - Duration of single invocation 125.10 ns 8.15 ns 0.07 0.15 True
SubtractBenchmark - Duration of single invocation 126.84 ns 8.18 ns 0.06 0.10 True
SubtractionOperatorBenchmark - Duration of single invocation 126.52 ns 8.56 ns 0.07 0.09 True
NegateBenchmark - Duration of single invocation 128.57 ns 6.26 ns 0.05 0.09 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.196601763451857 < 17.350576451936654.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 72.57735651135751 (T) = (0 -7.196342476490403) / Math.Sqrt((3.472773234521482 / (299)) + (0.40182612500891873 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6266088676329324 = (19.272933534522014 - 7.196342476490403) / 19.272933534522014 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.7617399357131 < 36.46752895970839.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 177.27867984087774 (T) = (0 -8.472479140287) / Math.Sqrt((2.866328158929843 / (299)) + (0.5373419404810963 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7867265146578285 = (39.72589057047543 - 8.472479140287) / 39.72589057047543 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.054492741901694 < 36.56800949831038.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 191.2298889569639 (T) = (0 -8.196478618235243) / Math.Sqrt((5.64081271202114 / (299)) + (0.21890132278363453 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7949774766573632 = (39.978430099297725 - 8.196478618235243) / 39.978430099297725 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.311694989918148 < 36.47409812107511.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 245.12821009846598 (T) = (0 -8.179579141482611) / Math.Sqrt((3.61836401689573 / (299)) + (0.11806622702718901 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7953811141517731 = (39.97470276301719 - 8.179579141482611) / 39.97470276301719 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.597912032398982 < 62.259571228447314.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 209.6145504902091 (T) = (0 -9.48312341912792) / Math.Sqrt((15.448350763246381 / (299)) + (0.5890596719119393 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8584102250312582 = (66.97604697247007 - 9.48312341912792) / 66.97604697247007 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.472464148012043 < 17.32497456485339.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 102.3108783367013 (T) = (0 -7.259355150969097) / Math.Sqrt((2.238215925256169 / (299)) + (0.14281726108191267 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6181949551638932 = (19.01325100113656 - 7.259355150969097) / 19.01325100113656 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.254289372255001 < 122.18408349445258.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 296.61774438781924 (T) = (0 -6.663240509371934) / Math.Sqrt((40.99368767916618 / (299)) + (1.0016203026794048 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9493341312458881 = (131.5133969518909 - 6.663240509371934) / 131.5133969518909 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.206380177892852 < 36.48648712378547.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 176.8223199929863 (T) = (0 -8.30037786304288) / Math.Sqrt((5.057109179619962 / (299)) + (0.3845950047741589 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.79289372249929 = (40.07786708934705 - 8.30037786304288) / 40.07786708934705 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.283864700004281 < 27.707233222154866.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 144.34889903185294 (T) = (0 -6.304100121052394) / Math.Sqrt((6.01003373647559 / (299)) + (0.18938015043976805 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7920673088624729 = (30.3179845678179 - 6.304100121052394) / 30.3179845678179 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.061499294617407 < 125.16386878359671.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 217.60148583978133 (T) = (0 -8.350376945116647) / Math.Sqrt((86.58317573166681 / (299)) + (0.7478952678578928 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9364248493320012 = (131.34655376160836 - 8.350376945116647) / 131.34655376160836 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.167611905480294 < 118.80037874561587.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 319.37995979084076 (T) = (0 -8.278756837323186) / Math.Sqrt((40.18556943662828 / (299)) + (0.3048329017032269 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9365910536342069 = (130.5613373476474 - 8.278756837323186) / 130.5613373476474 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.102822248242963 < 17.568444240791262.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 65.37385114565923 (T) = (0 -7.311970545737597) / Math.Sqrt((1.4189722384142354 / (299)) + (0.6774608995773216 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6147110969897415 = (18.9778903275159 - 7.311970545737597) / 18.9778903275159 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1737420552776272 < 48.693134303079624.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 209.7768658863642 (T) = (0 -1.5578863051048666) / Math.Sqrt((18.723848235947006 / (299)) + (0.10792205565936598 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9720970593014636 = (55.832334015840296 - 1.5578863051048666) / 55.832334015840296 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.109535043754741 < 118.64829359885533.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 283.69643757782006 (T) = (0 -8.409023357550868) / Math.Sqrt((45.68474050644227 / (299)) + (0.8649339899987368 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9359150699195907 = (131.2168609219018 - 8.409023357550868) / 131.2168609219018 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.249715892980744 < 36.508867889492166.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 110.69726287949933 (T) = (0 -8.583809597111143) / Math.Sqrt((5.110200643755788 / (299)) + (1.570506086360666 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.78473930802626 = (39.87634490257189 - 8.583809597111143) / 39.87634490257189 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.338797306572966 < 36.46594162222504.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 128.2301477556903 (T) = (0 -8.333702174040925) / Math.Sqrt((4.28090457256747 / (299)) + (1.1457245963075966 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7905149113753964 = (39.78183950350224 - 8.333702174040925) / 39.78183950350224 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.208522025648405 < 153.5448094494807.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 340.4030905874765 (T) = (0 -14.545393202242813) / Math.Sqrt((48.4336491156612 / (299)) + (0.5215720499259916 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9091505547419596 = (160.1043700479339 - 14.545393202242813) / 160.1043700479339 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.570289738958164 < 27.820769431318798.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 188.01804572521334 (T) = (0 -6.38670854000441) / Math.Sqrt((2.103912400512407 / (299)) + (0.2269942674808013 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7889080301483599 = (30.255573172646614 - 6.38670854000441) / 30.255573172646614 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.152266947108204 < 118.73036084231997.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 333.3286992611541 (T) = (0 -8.256068374273317) / Math.Sqrt((37.288985383128455 / (299)) + (0.2524847822551018 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9368045589796055 = (130.64341732513435 - 8.256068374273317) / 130.64341732513435 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.180380467947302 < 120.30113953223396.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 372.55529312407106 (T) = (0 -8.127090876823232) / Math.Sqrt((31.509679696011407 / (299)) + (0.10226071998472513 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9381471250975421 = (131.39390674466887 - 8.127090876823232) / 131.39390674466887 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.557861839828872 < 120.23690400441288.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 321.82584607541884 (T) = (0 -8.386667980801056) / Math.Sqrt((31.536805858556736 / (299)) + (1.0152841733454088 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9361707359839528 = (131.39220873191616 - 8.386667980801056) / 131.39220873191616 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.255490647952697 < 122.1607150520888.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 377.12823647955537 (T) = (0 -6.403110888916448) / Math.Sqrt((29.475377251741012 / (299)) + (0.2583232208169843 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9510700207609751 = (130.8627346363053 - 6.403110888916448) / 130.8627346363053 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 65.29 ns 9.54 ns 0.15 0.28 True
MultiplyBenchmark - Duration of single invocation 65.47 ns 8.35 ns 0.13 0.16 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.61 ns 8.70 ns 0.23 0.17 True
NegateBenchmark - Duration of single invocation 63.30 ns 6.39 ns 0.10 0.21 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.05 ns 6.22 ns 0.21 0.26 True
SubtractionOperatorBenchmark - Duration of single invocation 64.29 ns 8.03 ns 0.12 0.17 True
XorBenchmark - Duration of single invocation 40.47 ns 8.15 ns 0.20 0.21 True
MultiplyOperatorBenchmark - Duration of single invocation 64.44 ns 8.43 ns 0.13 0.17 True
AddOperatorBenchmark - Duration of single invocation 64.41 ns 7.54 ns 0.12 0.18 True
AddBenchmark - Duration of single invocation 64.58 ns 8.21 ns 0.13 0.18 True
ConditionalSelectBenchmark - Duration of single invocation 171.00 ns 14.74 ns 0.09 0.18 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.49 ns 7.79 ns 0.20 0.21 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.38 ns 8.22 ns 0.21 0.20 True
OnesComplementBenchmark - Duration of single invocation 28.52 ns 6.51 ns 0.23 0.20 True
BitwiseOrBenchmark - Duration of single invocation 38.49 ns 11.30 ns 0.29 0.19 True
SubtractBenchmark - Duration of single invocation 65.37 ns 8.16 ns 0.12 0.21 True
OneBenchmark - Duration of single invocation 29.74 ns 1.83 ns 0.06 0.22 True
BitwiseAndBenchmark - Duration of single invocation 38.10 ns 8.19 ns 0.21 0.16 True
EqualityOperatorBenchmark - Duration of single invocation 18.52 ns 7.15 ns 0.39 0.30 True
EqualsAllBenchmark - Duration of single invocation 18.10 ns 7.01 ns 0.39 0.30 True
InequalityOperatorBenchmark - Duration of single invocation 17.13 ns 7.06 ns 0.41 0.39 True
UnaryNegateOperatorBenchmark - Duration of single invocation 63.26 ns 7.51 ns 0.12 0.21 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.54156287448514 < 62.089306178764275.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 217.21941956750567 (T) = (0 -9.267438335103368) / Math.Sqrt((19.133471850554887 / (299)) + (0.16698706619293224 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8617070612284308 = (67.01309855314608 - 9.267438335103368) / 67.01309855314608 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.347678956159289 < 61.55173071867058.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 255.975429015394 (T) = (0 -8.287268098474112) / Math.Sqrt((3.974751615821194 / (299)) + (0.8325084285182995 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.8712312729912048 = (64.35776986370362 - 8.287268098474112) / 64.35776986370362 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.699113566869217 < 36.620559848772615.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 250.04898925694974 (T) = (0 -8.119644150287268) / Math.Sqrt((4.241196454744217 / (299)) + (0.04338435770654792 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7953173165376843 = (39.66942397343634 - 8.119644150287268) / 39.66942397343634 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.390064176545621 < 60.14474182269892.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 254.00845310909736 (T) = (0 -6.465419986334714) / Math.Sqrt((10.15826353598941 / (299)) + (0.4080525945198346 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.898072313737635 = (63.43144069504848 - 6.465419986334714) / 63.43144069504848 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.223848800809842 < 27.624789366290123.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 72.55716377861255 (T) = (0 -6.23804425658545) / Math.Sqrt((4.3936854790474476 / (299)) + (2.424979943230609 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7953874160765366 = (30.48709975198216 - 6.23804425658545) / 30.48709975198216 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.032514320479239 < 61.264964024330006.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 278.12008368290765 (T) = (0 -8.256349219406134) / Math.Sqrt((5.998246817224502 / (299)) + (0.503416722209251 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8710301188426517 = (64.0176539306341 - 8.256349219406134) / 64.0176539306341 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.150736430427157 < 38.126667683884925.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 236.717523338748 (T) = (0 -8.195588091168126) / Math.Sqrt((4.417189927868404 / (299)) + (0.0770245727562035 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7942135069976638 = (39.82568521188165 - 8.195588091168126) / 39.82568521188165 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.432466630677707 < 61.58490973178525.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 212.87929183368794 (T) = (0 -8.472325214743586) / Math.Sqrt((7.8823868449017365 / (299)) + (1.0833842262253477 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8689979190406654 = (64.67321093451594 - 8.472325214743586) / 64.67321093451594 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.543083024877759 < 61.38722026532076.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 222.7487131192622 (T) = (0 -8.358574639881404) / Math.Sqrt((13.549363096544802 / (299)) + (0.48935808527681046 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.871604998355033 = (65.10046756332632 - 8.358574639881404) / 65.10046756332632 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.208783268236768 < 61.49254764168489.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 255.86701544336313 (T) = (0 -8.145124534542173) / Math.Sqrt((11.58370107962971 / (299)) + (0.27028950592265294 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.874887571637191 = (65.10244138913544 - 8.145124534542173) / 65.10244138913544 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.737986807374886 < 153.3411153021434.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 436.9154035755635 (T) = (0 -14.60890151671797) / Math.Sqrt((28.43239671050507 / (299)) + (0.35281553181548025 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9081156404841786 = (158.99225497896066 - 14.60890151671797) / 158.99225497896066 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.78763514656685 < 36.57341463104255.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 187.55775653720823 (T) = (0 -8.334495290782133) / Math.Sqrt((3.3299617471905565 / (299)) + (0.42039426293670723 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7900230428572311 = (39.69242817970398 - 8.334495290782133) / 39.69242817970398 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.224817225456803 < 36.60543926343503.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 154.5440816514585 (T) = (0 -8.345022434963697) / Math.Sqrt((3.4665129197978914 / (299)) + (0.7562778778139974 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7911580978857725 = (39.95856363345767 - 8.345022434963697) / 39.95856363345767 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.509793851009842 < 27.623025143643762.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 198.79144400363114 (T) = (0 -6.342991311471931) / Math.Sqrt((2.4829660721968794 / (299)) + (0.15410586489345054 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7903462874193664 = (30.25461001093597 - 6.342991311471931) / 30.25461001093597 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.303923258587151 < 36.43059615602794.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 123.55494561303834 (T) = (0 -8.29368855722074) / Math.Sqrt((6.17775396331495 / (299)) + (1.1362111468958318 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7929783000373848 = (40.0619285742434 - 8.29368855722074) / 40.0619285742434 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.163970777412315 < 61.31392020245854.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 291.10449830506826 (T) = (0 -8.227905381552228) / Math.Sqrt((6.968834806635034 / (299)) + (0.3407169210422785 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8717883265254767 = (64.17438567469578 - 8.227905381552228) / 64.17438567469578 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.8318316304881328 < 28.212969484218345.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 226.65386725959192 (T) = (0 -1.5764606847115936) / Math.Sqrt((3.0299988005901812 / (299)) + (0.18938988484619332 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.950330138435917 = (31.73877750147696 - 1.5764606847115936) / 31.73877750147696 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.18677893116063 < 36.408327622436445.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 110.4587761661274 (T) = (0 -8.43367808045182) / Math.Sqrt((2.051536684558089 / (299)) + (1.8065427659948015 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7865130818700201 = (39.50442563097492 - 8.43367808045182) / 39.50442563097492 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.148837735860312 < 17.293272065542993.
IsChangePoint: Marked as a change because one of 4/11/2023 3:40:00 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 52.552200831713684 (T) = (0 -7.549957750273489) / Math.Sqrt((2.3262524146655665 / (299)) + (1.0074557355795248 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6041538431668326 = (19.072959582769123 - 7.549957750273489) / 19.072959582769123 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.01463281725055 < 17.488536275733356.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 53.601602342925496 (T) = (0 -7.234725012760241) / Math.Sqrt((1.5281919204114953 / (299)) + (1.0937826675026008 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6208863930622918 = (19.08326390919799 - 7.234725012760241) / 19.08326390919799 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.055254007539861 < 17.277049411717826.
IsChangePoint: Marked as a change because one of 4/12/2023 4:47:47 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 71.02423526115219 (T) = (0 -7.392615364482626) / Math.Sqrt((3.9556792362249786 / (299)) + (0.3557514345192517 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.6167329017803582 = (19.288416351998166 - 7.392615364482626) / 19.288416351998166 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.510537635580038 < 60.22396774593977.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 260.4235015036834 (T) = (0 -6.596658110578181) / Math.Sqrt((8.377192522394273 / (299)) + (0.4792496627075917 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8955688685031542 = (63.167544160693346 - 6.596658110578181) / 63.167544160693346 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 38.37 ns 8.18 ns 0.21 0.20 True
MultiplyOperatorBenchmark - Duration of single invocation 81.78 ns 8.22 ns 0.10 0.22 True
DivisionOperatorBenchmark - Duration of single invocation 83.12 ns 8.11 ns 0.10 0.18 True
SubtractionOperatorBenchmark - Duration of single invocation 82.37 ns 8.24 ns 0.10 0.15 True
AddOperatorBenchmark - Duration of single invocation 81.51 ns 8.03 ns 0.10 0.27 True
AndNotBenchmark - Duration of single invocation 65.84 ns 10.44 ns 0.16 0.20 True
SubtractBenchmark - Duration of single invocation 82.90 ns 8.19 ns 0.10 0.19 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.36 ns 8.19 ns 0.21 0.26 True
NegateBenchmark - Duration of single invocation 86.43 ns 7.27 ns 0.08 0.18 True
BitwiseOrBenchmark - Duration of single invocation 38.18 ns 8.53 ns 0.22 0.28 True
UnaryNegateOperatorBenchmark - Duration of single invocation 82.70 ns 7.34 ns 0.09 0.16 True
ConditionalSelectBenchmark - Duration of single invocation 163.22 ns 15.10 ns 0.09 0.20 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.45 ns 8.43 ns 0.22 0.21 True
MultiplyBenchmark - Duration of single invocation 88.76 ns 8.11 ns 0.09 0.16 True
DivideBenchmark - Duration of single invocation 83.59 ns 7.97 ns 0.10 0.20 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.77 ns 10.73 ns 0.28 0.25 True
AddBenchmark - Duration of single invocation 82.59 ns 8.15 ns 0.10 0.21 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.02 ns 5.57 ns 0.19 0.21 True
XorBenchmark - Duration of single invocation 41.16 ns 8.26 ns 0.20 0.15 True
OnesComplementBenchmark - Duration of single invocation 29.18 ns 6.19 ns 0.21 0.17 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.182962922986135 < 36.41483351424294.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 160.51169690697168 (T) = (0 -8.233554478442535) / Math.Sqrt((4.5383207634159985 / (299)) + (0.5900958752742896 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7933522396480501 = (39.843424697270606 - 8.233554478442535) / 39.843424697270606 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.221943100170886 < 77.95421741864963.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 231.33801909834932 (T) = (0 -8.550710204309125) / Math.Sqrt((17.260781745955594 / (299)) + (0.9799813768723631 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.893877078577596 = (80.57364129917323 - 8.550710204309125) / 80.57364129917323 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.109214138969776 < 79.43336085037829.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 306.83360317236384 (T) = (0 -8.334972689596286) / Math.Sqrt((14.003894819855667 / (299)) + (0.23741501710476665 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8973022329886732 = (81.16021343168055 - 8.334972689596286) / 81.16021343168055 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.244801324382319 < 78.33136585038827.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 296.7901216144949 (T) = (0 -8.357235414877085) / Math.Sqrt((9.110784214787495 / (299)) + (0.6908283559248601 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.895400939775473 = (79.89780593571172 - 8.357235414877085) / 79.89780593571172 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.025065063645696 < 78.36501296356026.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 276.75990894622663 (T) = (0 -8.16401197541733) / Math.Sqrt((19.557444200472407 / (299)) + (0.10924200577905542 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8995473262119196 = (81.2722217095037 - 8.16401197541733) / 81.2722217095037 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.441324986084 < 62.06382915831954.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 229.62956609057676 (T) = (0 -9.48167998796271) / Math.Sqrt((14.127045403111152 / (299)) + (0.37024149682295304 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8578147494007826 = (66.68539773291293 - 9.48167998796271) / 66.68539773291293 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.189360353154392 < 78.42928811011882.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 293.43554791336356 (T) = (0 -8.341661016211566) / Math.Sqrt((8.853020825222806 / (299)) + (0.718352629984694 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.8956541277909538 = (79.94241496683165 - 8.341661016211566) / 79.94241496683165 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.18585891003741 < 36.464409836278364.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 162.46794630607695 (T) = (0 -8.085712903829796) / Math.Sqrt((5.332673525213915 / (299)) + (0.5299932722847954 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7987870648552897 = (40.1848563961091 - 8.085712903829796) / 40.1848563961091 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.268323072082157 < 78.23845619382266.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 213.1549657871255 (T) = (0 -7.6925237115053315) / Math.Sqrt((29.66794188015741 / (299)) + (0.33936614082192884 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9029722279436312 = (79.28166903632824 - 7.6925237115053315) / 79.28166903632824 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.534883887744407 < 36.41975718257719.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 144.72560289124755 (T) = (0 -8.503426839708997) / Math.Sqrt((7.297344065533951 / (299)) + (0.601514458724119 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7893360901333696 = (40.36489612811442 - 8.503426839708997) / 40.36489612811442 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.340769792541557 < 78.22033011640613.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 263.2437040567017 (T) = (0 -7.653279499284138) / Math.Sqrt((12.123376832979888 / (299)) + (0.802923225406947 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.903445775819363 = (79.2640566917727 - 7.653279499284138) / 79.2640566917727 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.104362308466735 < 153.3559951464837.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 371.66737770191975 (T) = (0 -14.686756553122041) / Math.Sqrt((40.08452296958083 / (299)) + (0.4504709824857151 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9079937878682165 = (159.62787960540885 - 14.686756553122041) / 159.62787960540885 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.433022981821443 < 36.49581202084908.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 210.38467452110092 (T) = (0 -8.117163190378138) / Math.Sqrt((4.9279214188933995 / (299)) + (0.15974012726176137 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7967352279848774 = (39.933939904619734 - 8.117163190378138) / 39.933939904619734 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.111661560158561 < 77.92937812794021.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 334.0796042453821 (T) = (0 -8.305328547731028) / Math.Sqrt((8.660009121302084 / (299)) + (0.4288199238140198 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8962453060009352 = (80.04773786722248 - 8.305328547731028) / 80.04773786722248 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.970371707177508 < 79.39750690426523.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 273.5173884496063 (T) = (0 -8.44328044570423) / Math.Sqrt((11.782938941569634 / (299)) + (0.7776903883379243 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8958585516997299 = (81.07512026681057 - 8.44328044570423) / 81.07512026681057 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.733390295977362 < 37.21109490951789.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 111.09071709612087 (T) = (0 -8.496466512648878) / Math.Sqrt((6.678213301852313 / (299)) + (1.4553308958711275 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7877219555020754 = (40.025177981757544 - 8.496466512648878) / 40.025177981757544 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.150420928044413 < 78.40184518711555.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 296.570680266759 (T) = (0 -8.394388540972226) / Math.Sqrt((10.966488405873553 / (299)) + (0.5744480828902294 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8961476171158065 = (80.83000416401485 - 8.394388540972226) / 80.83000416401485 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.571198458432941 < 27.57242189976239.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 263.8046898801575 (T) = (0 -6.334308701055851) / Math.Sqrt((1.519161915424279 / (299)) + (0.077576910524174 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7902502002421788 = (30.19935517635533 - 6.334308701055851) / 30.19935517635533 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.263675043544165 < 36.53946207777464.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 149.56669230593798 (T) = (0 -8.325774384147815) / Math.Sqrt((3.519184295758136 / (299)) + (0.8043480784610781 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7901722725839049 = (39.67909525911958 - 8.325774384147815) / 39.67909525911958 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Single>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.194068052261666 < 27.64495390981157.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 138.09307191867774 (T) = (0 -6.51996877269294) / Math.Sqrt((4.18990335023245 / (299)) + (0.3904715804978827 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7847570366127967 = (30.291205204065545 - 6.51996877269294) / 30.291205204065545 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 38.50 ns 8.16 ns 0.21 0.20 True
AddBenchmark - Duration of single invocation 202.31 ns 8.27 ns 0.04 0.09 True
BitwiseOrOperatorBenchmark - Duration of single invocation 46.23 ns 8.23 ns 0.18 0.25 True
MultiplyBenchmark - Duration of single invocation 202.23 ns 8.46 ns 0.04 0.08 True
UnaryNegateOperatorBenchmark - Duration of single invocation 209.65 ns 6.26 ns 0.03 0.11 True
SubtractBenchmark - Duration of single invocation 207.71 ns 8.23 ns 0.04 0.10 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.55 ns 6.29 ns 0.21 0.22 True
BitwiseOrBenchmark - Duration of single invocation 38.56 ns 8.21 ns 0.21 0.22 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.41 ns 6.21 ns 0.16 0.19 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.41 ns 8.22 ns 0.21 0.24 True
OneBenchmark - Duration of single invocation 76.33 ns 1.76 ns 0.02 0.23 True
EqualsAllBenchmark - Duration of single invocation 16.61 ns 6.92 ns 0.42 0.28 True
AddOperatorBenchmark - Duration of single invocation 208.69 ns 6.93 ns 0.03 0.13 True
XorBenchmark - Duration of single invocation 44.24 ns 8.40 ns 0.19 0.22 True
EqualityOperatorBenchmark - Duration of single invocation 18.81 ns 7.70 ns 0.41 0.30 True
AndNotBenchmark - Duration of single invocation 66.55 ns 9.61 ns 0.14 0.25 True
ConditionalSelectBenchmark - Duration of single invocation 161.89 ns 15.18 ns 0.09 0.21 True
MultiplyOperatorBenchmark - Duration of single invocation 216.13 ns 9.44 ns 0.04 0.13 True
SubtractionOperatorBenchmark - Duration of single invocation 209.81 ns 9.05 ns 0.04 0.11 True
NegateBenchmark - Duration of single invocation 209.01 ns 8.42 ns 0.04 0.11 True
OnesComplementBenchmark - Duration of single invocation 28.83 ns 7.18 ns 0.25 0.17 True
InequalityOperatorBenchmark - Duration of single invocation 16.34 ns 6.32 ns 0.39 0.28 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.158058016669743 < 36.50288715337437.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 204.82858530124918 (T) = (0 -8.235083125191608) / Math.Sqrt((5.148240339228922 / (299)) + (0.16769673046895264 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7937003992251436 = (39.91807591609886 - 8.235083125191608) / 39.91807591609886 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.267057776269736 < 192.5502632262136.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 397.73889505677533 (T) = (0 -8.209855804290463) / Math.Sqrt((73.77791012049394 / (299)) + (0.19854627576508518 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9607064195755762 = (208.9363126396969 - 8.209855804290463) / 208.9363126396969 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.225929301522497 < 37.043760678152864.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 203.78547691748216 (T) = (0 -8.170890964178485) / Math.Sqrt((6.856196753601649 / (299)) + (0.04851187122083142 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7972931302763616 = (40.3089001143292 - 8.170890964178485) / 40.3089001143292 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.457556320747578 < 192.24051536162037.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 507.6316704851295 (T) = (0 -8.49572552827288) / Math.Sqrt((45.139939842331685 / (299)) + (0.10287921285235718 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9592345837579832 = (208.4052197047446 - 8.49572552827288) / 208.4052197047446 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.260127147008378 < 198.79567250478263.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 364.28485821101054 (T) = (0 -6.4232536128236255) / Math.Sqrt((92.30905235418415 / (299)) + (0.34196509369683553 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9698815862202762 = (213.26666337082682 - 6.4232536128236255) / 213.26666337082682 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.230492542665074 < 196.5861216354473.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 451.24660068590947 (T) = (0 -8.217780049809841) / Math.Sqrt((59.65536160940131 / (299)) + (0.1658634969622428 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9614371641420397 = (213.1010302270987 - 8.217780049809841) / 213.1010302270987 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.288244975282516 < 28.406048558377634.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 192.6031102942222 (T) = (0 -6.303749519268468) / Math.Sqrt((2.5552549999948315 / (299)) + (0.17400862043225954 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7918699649893185 = (30.287553254603313 - 6.303749519268468) / 30.287553254603313 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.209510760465111 < 36.60321935071658.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 238.7034649847066 (T) = (0 -8.14528865279714) / Math.Sqrt((4.412271757271911 / (299)) + (0.07763139942591082 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7966105013934459 = (40.04773456152601 - 8.14528865279714) / 40.04773456152601 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.208435501062555 < 36.491638080711624.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 142.33461310300817 (T) = (0 -8.230866605126709) / Math.Sqrt((5.659445742288126 / (299)) + (0.759531566193517 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7933875257467271 = (39.83722006560464 - 8.230866605126709) / 39.83722006560464 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.216396442830806 < 36.52959048844031.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 165.09550981161152 (T) = (0 -8.257904941541451) / Math.Sqrt((4.427789910654544 / (299)) + (0.548129265918545 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7930361549861701 = (39.90022963184528 - 8.257904941541451) / 39.90022963184528 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.7576964587383552 < 72.53429185248879.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 170.35998126505368 (T) = (0 -1.796336010372639) / Math.Sqrt((70.23132244888873 / (299)) + (0.13504993791583875 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.978942353188931 = (85.30563868268447 - 1.796336010372639) / 85.30563868268447 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.918135439510773 < 16.478172901877926.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 89.33123902822207 (T) = (0 -7.252105241592724) / Math.Sqrt((3.0671967660111545 / (299)) + (0.18728875121848024 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6213699537676651 = (19.153538694979023 - 7.252105241592724) / 19.153538694979023 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.932868026096444 < 193.40078937756465.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 385.94825236562406 (T) = (0 -8.33235961695381) / Math.Sqrt((66.03916358326984 / (299)) + (1.244941789551785 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9601558066410661 = (209.12356141564385 - 8.33235961695381) / 209.12356141564385 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.403147583413347 < 36.44652037332616.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 257.36071515262796 (T) = (0 -8.151197040472221) / Math.Sqrt((3.8331857386381993 / (299)) + (0.05833532491159632 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7953611803255622 = (39.832115203948355 - 8.151197040472221) / 39.832115203948355 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.698801759158497 < 16.80129643404885.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 86.00394509933744 (T) = (0 -7.07432597216026) / Math.Sqrt((2.7327126181598116 / (299)) + (0.2612412411354975 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6298402278202351 = (19.11154723945711 - 7.07432597216026) / 19.11154723945711 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.612654933634222 < 62.44867266111035.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 256.7438197298813 (T) = (0 -9.292743677356691) / Math.Sqrt((9.856774126563584 / (299)) + (0.42407251773294935 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8605979567509459 = (66.6614596226138 - 9.292743677356691) / 66.6614596226138 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.175147579272076 < 153.79842822232658.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 318.8333363631429 (T) = (0 -15.01375640995996) / Math.Sqrt((39.62702183426825 / (299)) + (1.8318512823238606 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9059613462969702 = (159.65516113589618 - 15.01375640995996) / 159.65516113589618 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.440577041401003 < 192.08803623189934.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 354.4412890234688 (T) = (0 -8.629292255921696) / Math.Sqrt((91.26008449766027 / (299)) + (0.40385700107173844 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.958822034099629 = (209.5609160685599 - 8.629292255921696) / 209.5609160685599 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.04973445587611 < 196.73832745312487.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 414.80720797588947 (T) = (0 -8.390087347337587) / Math.Sqrt((65.63530258448391 / (299)) + (0.5979381900290356 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9606194905480987 = (213.05177266904352 - 8.390087347337587) / 213.05177266904352 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.420183617018118 < 198.31506672042931.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 347.8621698384193 (T) = (0 -6.773321381790302) / Math.Sqrt((92.6573306238272 / (299)) + (1.0372602786440497 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9681969365406963 = (212.97701054672524 - 6.773321381790302) / 212.97701054672524 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.1841263313085095 < 27.65922530394966.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 159.82131671829302 (T) = (0 -6.468371280039147) / Math.Sqrt((1.5613959535478183 / (299)) + (0.41716344928985966 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7852780662882052 = (30.124408662978777 - 6.468371280039147) / 30.124408662978777 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<SByte>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.316292863797085 < 16.752480888318.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 62.40587106545721 (T) = (0 -7.264030531013162) / Math.Sqrt((3.382218288030772 / (299)) + (0.6173896321031676 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6198021281742985 = (19.10592107244434 - 7.264030531013162) / 19.10592107244434 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 122.17 ns 11.23 ns 0.09 0.14 True
AddOperatorBenchmark - Duration of single invocation 123.59 ns 8.34 ns 0.07 0.15 True
SubtractionOperatorBenchmark - Duration of single invocation 119.57 ns 8.62 ns 0.07 0.13 True
SubtractBenchmark - Duration of single invocation 123.21 ns 8.15 ns 0.07 0.12 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.231080958482838 < 115.93468921733471.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 399.6598698025872 (T) = (0 -8.302497224433322) / Math.Sqrt((18.01376409766451 / (299)) + (0.6556805672992377 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.934017236529815 = (125.82827374583825 - 8.302497224433322) / 125.82827374583825 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.34325560191878 < 113.81256380846526.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 511.79671715198265 (T) = (0 -8.198045706968314) / Math.Sqrt((12.0432373609588 / (299)) + (0.24463468257200036 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9331929206042393 = (122.7122302174541 - 8.198045706968314) / 122.7122302174541 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.622841291183471 < 113.77253827721835.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 494.7990527298106 (T) = (0 -8.104738237763563) / Math.Sqrt((12.65219352851911 / (299)) + (0.2800799853429545 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9338775117176412 = (122.57158567830069 - 8.104738237763563) / 122.57158567830069 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.150183379742128 < 116.47470383803818.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 452.06646890679724 (T) = (0 -8.248794383440616) / Math.Sqrt((12.077661559732594 / (299)) + (0.6626250725207369 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9341017170414312 = (125.17464815626136 - 8.248794383440616) / 125.17464815626136 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 65.43 ns 9.47 ns 0.14 0.23 True
XorBenchmark - Duration of single invocation 40.39 ns 8.02 ns 0.20 0.14 True
BitwiseOrBenchmark - Duration of single invocation 38.70 ns 8.10 ns 0.21 0.26 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.31 ns 8.26 ns 0.22 0.22 True
ConditionalSelectBenchmark - Duration of single invocation 170.86 ns 15.25 ns 0.09 0.20 True
BitwiseOrOperatorBenchmark - Duration of single invocation 39.88 ns 8.12 ns 0.20 0.18 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.41 ns 8.39 ns 0.22 0.23 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.10 ns 6.25 ns 0.21 0.19 True
BitwiseAndBenchmark - Duration of single invocation 38.30 ns 8.35 ns 0.22 0.23 True
OnesComplementBenchmark - Duration of single invocation 29.59 ns 6.36 ns 0.21 0.23 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Double>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.472028526733373 < 62.17117574377293.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 275.205178710205 (T) = (0 -9.462304633356105) / Math.Sqrt((8.049757835908995 / (299)) + (0.4011447743033945 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8577286710062325 = (66.50886514014795 - 9.462304633356105) / 66.50886514014795 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.02058933441033 < 36.65326107613649.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 280.7691450925954 (T) = (0 -8.080971562110223) / Math.Sqrt((3.257426487196708 / (299)) + (0.04297967888919676 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7960083222606402 = (39.61422177445534 - 8.080971562110223) / 39.61422177445534 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.099288853510195 < 36.4817447975625.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 132.77887588514164 (T) = (0 -8.317223830089976) / Math.Sqrt((6.805171374971144 / (299)) + (0.8868488573175531 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7939188646222636 = (40.35897713221017 - 8.317223830089976) / 40.35897713221017 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.255184876436582 < 36.43165448128217.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 124.52198130961798 (T) = (0 -8.364413961241443) / Math.Sqrt((11.190270727366272 / (299)) + (0.6715547471093616 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.7919852488009029 = (40.2106769497112 - 8.364413961241443) / 40.2106769497112 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.248829275341475 < 153.94917625839395.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 369.4422474723804 (T) = (0 -14.72172446364811) / Math.Sqrt((38.65090959318472 / (299)) + (0.6131276006963647 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9077606759328541 = (159.60355968059122 - 14.72172446364811) / 159.60355968059122 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.124154173413944 < 36.47256041208264.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 148.54383769771883 (T) = (0 -8.365795765859668) / Math.Sqrt((5.4536483429708085 / (299)) + (0.6829831984117667 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7912298487498094 = (40.07180009097219 - 8.365795765859668) / 40.07180009097219 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.392544837948307 < 36.468379591632775.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 208.72025241850432 (T) = (0 -8.222112637783013) / Math.Sqrt((4.068055002405916 / (299)) + (0.23224163480704615 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7934345389046055 = (39.8039081373141 - 8.222112637783013) / 39.8039081373141 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.250946608609373 < 27.800092609606637.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 221.6980130989625 (T) = (0 -6.305747199076824) / Math.Sqrt((3.0687662153300566 / (299)) + (0.03630133439369012 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7919027049371391 = (30.30191813484177 - 6.305747199076824) / 30.30191813484177 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.345192639875311 < 36.71720448949366.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 120.63315347000402 (T) = (0 -8.442210048732422) / Math.Sqrt((6.76254157494605 / (299)) + (1.149205770060638 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7891272005347455 = (40.03460887388392 - 8.442210048732422) / 40.03460887388392 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Double>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.359425088025584 < 28.15527092082836.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 229.7993487963043 (T) = (0 -6.155422266002582) / Math.Sqrt((2.079107811467609 / (299)) + (0.1011320550005228 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7965532250160566 = (30.255688577459075 - 6.155422266002582) / 30.255688577459075 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Narrow_ushort - Duration of single invocation 201.75 μs 158.58 μs 0.79 0.19 False
Widen_short - Duration of single invocation 176.65 μs 106.30 μs 0.60 0.26 True
Widen_sbyte - Duration of single invocation 244.18 μs 179.39 μs 0.73 0.12 True
Narrow_double - Duration of single invocation 103.17 μs 71.01 μs 0.69 0.30 True
Widen_ushort - Duration of single invocation 178.14 μs 108.09 μs 0.61 0.19 True
Widen_uint - Duration of single invocation 149.52 μs 82.63 μs 0.55 0.19 True
Widen_byte - Duration of single invocation 255.08 μs 179.60 μs 0.70 0.17 True
Convert_int_float - Duration of single invocation 83.37 μs 54.13 μs 0.65 0.22 True
Convert_float_uint - Duration of single invocation 108.97 μs 68.26 μs 0.63 0.22 True
Convert_double_ulong - Duration of single invocation 69.03 μs 39.24 μs 0.57 0.24 True
Narrow_int - Duration of single invocation 129.65 μs 101.01 μs 0.78 0.27 True
Widen_float - Duration of single invocation 155.81 μs 85.24 μs 0.55 0.26 True
Narrow_ulong - Duration of single invocation 88.10 μs 57.49 μs 0.65 0.19 True
Convert_double_long - Duration of single invocation 66.93 μs 35.82 μs 0.54 0.19 True
Convert_long_double - Duration of single invocation 109.52 μs 77.99 μs 0.71 0.19 True
Narrow_uint - Duration of single invocation 129.52 μs 100.26 μs 0.77 0.28 True
Narrow_long - Duration of single invocation 87.51 μs 57.44 μs 0.66 0.26 True
Convert_uint_float - Duration of single invocation 126.94 μs 75.48 μs 0.59 0.19 True
Widen_int - Duration of single invocation 146.93 μs 73.56 μs 0.50 0.24 True
Convert_ulong_double - Duration of single invocation 108.10 μs 76.53 μs 0.71 0.23 True
Narrow_short - Duration of single invocation 198.83 μs 158.25 μs 0.80 0.18 False
Convert_float_int - Duration of single invocation 97.96 μs 59.02 μs 0.60 0.17 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorConvert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorConvert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorConvert.Narrow_ushort

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 158.5808377946128 < 185.28589232808807.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 34.67946593219658 (T) = (0 -160522.37278932508) / Math.Sqrt((90895610.57378419 / (299)) + (27026350.00234561 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.202714139002208 = (201336.03346287063 - 160522.37278932508) / 201336.03346287063 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_short

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 106.30163274872451 < 168.09954915267315.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 92.75827308119607 (T) = (0 -106578.22656761741) / Math.Sqrt((42197996.09439416 / (299)) + (14578373.044035355 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.425513219537592 = (185519.02357410546 - 106578.22656761741) / 185519.02357410546 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 179.39399284699678 < 231.99393045134715.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 52.35718398046223 (T) = (0 -176957.07293995668) / Math.Sqrt((128096246.5890904 / (299)) + (48521901.93336609 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3129158407699818 = (257547.88632918545 - 176957.07293995668) / 257547.88632918545 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 71.00656020359848 < 97.97591703404015.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 50.808815745105065 (T) = (0 -71851.37231369756) / Math.Sqrt((20758492.068680592 / (299)) + (9404094.540906684 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3206671264871457 = (105767.54800949286 - 71851.37231369756) / 105767.54800949286 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 108.08858117816092 < 169.2808821919674.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 92.41872967396895 (T) = (0 -107908.53398837523) / Math.Sqrt((46214367.73057903 / (299)) + (13292721.077415528 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.4188955780808853 = (185695.59947935704 - 107908.53398837523) / 185695.59947935704 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_uint

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 82.63475454545454 < 138.99007417341159.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 83.01257109784481 (T) = (0 -74717.32051272757) / Math.Sqrt((24581082.776416842 / (299)) + (19174523.303769138 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.505885999412787 = (151214.74077628303 - 74717.32051272757) / 151214.74077628303 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_byte

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 179.59826400162342 < 240.32331361692482.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 70.17258444923715 (T) = (0 -176408.4085227215) / Math.Sqrt((97892173.75351937 / (299)) + (25605460.808029275 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3162210322259133 = (257990.39870586508 - 176408.4085227215) / 257990.39870586508 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 54.13452813082246 < 79.21510816422081.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 48.71609940101219 (T) = (0 -53275.73157354796) / Math.Sqrt((35429256.698307514 / (299)) + (9287597.262163142 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.39027653148215297 = (87376.87545971267 - 53275.73157354796) / 87376.87545971267 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_float_uint

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 68.2576068959243 < 94.31451621073995.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 50.64572947165687 (T) = (0 -68813.23998392145) / Math.Sqrt((19525181.904408213 / (299)) + (10104464.027080936 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.33523409858077036 = (103514.99653789387 - 68813.23998392145) / 103514.99653789387 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 39.23518911559467 < 65.51877346294032.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 82.3070325891228 (T) = (0 -37811.95135118654) / Math.Sqrt((20819328.95731709 / (299)) + (2594517.1216377732 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.47546587734079704 = (72086.73319381641 - 37811.95135118654) / 72086.73319381641 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 101.00824792183624 < 128.84602976765424.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 41.633377995454296 (T) = (0 -100654.8977691351) / Math.Sqrt((27495046.377939012 / (299)) + (16237267.490588479 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.26262466451652716 = (136504.291539856 - 100654.8977691351) / 136504.291539856 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_float

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 85.23925833333332 < 157.02254058964644.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 151.80253441079998 (T) = (0 -82728.86312701213) / Math.Sqrt((62058516.00959124 / (299)) + (1847784.4220621604 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.49328498378304125 = (163265.0710544373 - 82728.86312701213) / 163265.0710544373 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 57.491822150735295 < 83.19572792726622.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 123.84183808358706 (T) = (0 -56514.11342575806) / Math.Sqrt((15807048.71940124 / (299)) + (702096.0867399228 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.38403757914908104 = (91749.28780182215 - 56514.11342575806) / 91749.28780182215 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 35.81558729500381 < 63.42207612179487.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 149.21855073429796 (T) = (0 -35262.81282932084) / Math.Sqrt((13340410.642071137 / (299)) + (226809.02331389405 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.4950778101436383 = (69838.11275822968 - 35262.81282932084) / 69838.11275822968 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_long_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 77.99245829187396 < 107.90025758290443.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 71.21916541938694 (T) = (0 -78473.8414788233) / Math.Sqrt((16384716.920715041 / (299)) + (4422892.361582254 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.30404084477675464 = (112756.38935111208 - 78473.8414788233) / 112756.38935111208 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_uint

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 100.26179679487178 < 125.73287677772038.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 60.16073556496886 (T) = (0 -99295.12315501733) / Math.Sqrt((44886630.26667629 / (299)) + (5985726.1282514995 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.2743912236965247 = (136843.88391891305 - 99295.12315501733) / 136843.88391891305 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_long

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 57.43697830116422 < 83.21968552699757.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 72.61672059210235 (T) = (0 -56915.67838152631) / Math.Sqrt((20758914.087371394 / (299)) + (4152673.7380297356 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3824117569759319 = (92157.96936618212 - 56915.67838152631) / 92157.96936618212 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_uint_float

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 75.47853686594202 < 101.00390364689625.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 58.5477629807948 (T) = (0 -73716.16193224385) / Math.Sqrt((15991584.996718096 / (299)) + (6369137.051193867 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3060178501012422 = (106221.98559861806 - 73716.16193224385) / 106221.98559861806 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Widen_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 73.56190203722335 < 140.17807734723453.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 107.41707151726251 (T) = (0 -73815.49700478652) / Math.Sqrt((38037958.58598505 / (299)) + (10113815.762236187 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.5148401923263969 = (152146.76862607454 - 73815.49700478652) / 152146.76862607454 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 76.52941880056633 < 103.85820074043217.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 49.58022198870739 (T) = (0 -77424.6147784995) / Math.Sqrt((20900315.755571786 / (299)) + (11168059.507355299 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3151969522812204 = (113061.14223121069 - 77424.6147784995) / 113061.14223121069 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Narrow_short

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 158.24535618686872 < 180.5432635011198.
IsChangePoint: Marked as a change because one of 5/10/2023 2:00:36 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 30.5695285310085 (T) = (0 -159617.1090030145) / Math.Sqrt((79874308.35144293 / (299)) + (37204468.8955295 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.20238576577617468 = (200118.18013546508 - 159617.1090030145) / 200118.18013546508 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 59.016350698476046 < 84.81028923034529.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 80.65391091609389 (T) = (0 -58177.14846764508) / Math.Sqrt((16679082.06817114 / (299)) + (3309348.232599125 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.3755299548005076 = (93162.43255360612 - 58177.14846764508) / 93162.43255360612 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OneBenchmark - Duration of single invocation 19.48 ns 1.33 ns 0.07 0.28 True
ConditionalSelectBenchmark - Duration of single invocation 161.52 ns 14.96 ns 0.09 0.18 True
XorBenchmark - Duration of single invocation 39.15 ns 8.47 ns 0.22 0.24 True
InequalityOperatorBenchmark - Duration of single invocation 18.20 ns 8.46 ns 0.46 0.26 True
BitwiseAndBenchmark - Duration of single invocation 38.28 ns 8.41 ns 0.22 0.19 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.85 ns 6.16 ns 0.21 0.20 True
EqualityOperatorBenchmark - Duration of single invocation 18.14 ns 7.07 ns 0.39 0.26 True
OnesComplementBenchmark - Duration of single invocation 32.04 ns 6.33 ns 0.20 0.16 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.34 ns 8.01 ns 0.21 0.23 True
AndNotBenchmark - Duration of single invocation 75.09 ns 9.51 ns 0.13 0.24 True
BitwiseOrBenchmark - Duration of single invocation 38.40 ns 8.03 ns 0.21 0.20 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.75 ns 8.84 ns 0.23 0.20 True
BitwiseAndOperatorBenchmark - Duration of single invocation 38.66 ns 8.99 ns 0.23 0.17 True
EqualsAllBenchmark - Duration of single invocation 18.28 ns 7.17 ns 0.39 0.35 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.328927165752994 < 18.38066646509562.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 216.16603230784634 (T) = (0 -1.54493768378332) / Math.Sqrt((0.9035528160662324 / (299)) + (0.10745285728429947 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9229053491249697 = (20.0394925750121 - 1.54493768378332) / 20.0394925750121 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.96003783771915 < 153.37216509003062.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 382.5545875603036 (T) = (0 -14.749512545169061) / Math.Sqrt((34.67244093514745 / (299)) + (0.6700533308025217 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9074070640210253 = (159.29414473387337 - 14.749512545169061) / 159.29414473387337 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.468185494857801 < 36.84907627984776.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 182.8345685675553 (T) = (0 -8.135078685590477) / Math.Sqrt((5.30604673047699 / (299)) + (0.3107890124252301 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7960955315363372 = (39.896519908979855 - 8.135078685590477) / 39.896519908979855 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.4595762122206 < 17.21782607165487.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 81.74600413679174 (T) = (0 -7.2540403767622585) / Math.Sqrt((2.880073129103324 / (299)) + (0.2959261533042428 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6228105886978075 = (19.23182401043211 - 7.2540403767622585) / 19.23182401043211 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.40504628973637 < 36.44331662923538.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 126.88307082926683 (T) = (0 -8.55286477450246) / Math.Sqrt((3.0229747298973932 / (299)) + (1.2500897859443958 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7843573991377015 = (39.66222230812366 - 8.55286477450246) / 39.66222230812366 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.1556726960571515 < 28.126609366486864.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 164.60116919584942 (T) = (0 -6.345003289287038) / Math.Sqrt((2.833008249739066 / (299)) + (0.29197205855700237 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7904914902571706 = (30.28518171923182 - 6.345003289287038) / 30.28518171923182 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.0680222459633395 < 17.743994990365255.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 69.17010306408814 (T) = (0 -7.3171110502470285) / Math.Sqrt((1.8693901718442987 / (299)) + (0.577707650814269 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6182907738124085 = (19.169332434870263 - 7.3171110502470285) / 19.169332434870263 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.326383555448168 < 27.891682796254642.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 117.81250930332597 (T) = (0 -6.475325429145504) / Math.Sqrt((1.9371901436931687 / (299)) + (0.849649574495032 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7854030094777048 = (30.174353393239972 - 6.475325429145504) / 30.174353393239972 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.014037886923441 < 36.52532734643799.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 143.69270090354735 (T) = (0 -8.320692622989322) / Math.Sqrt((4.116834981626746 / (299)) + (0.852638339318751 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7907318322020945 = (39.76090922258555 - 8.320692622989322) / 39.76090922258555 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.50577104085418 < 62.12125011504286.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 246.60204776360564 (T) = (0 -9.248238994983579) / Math.Sqrt((11.655315250735613 / (299)) + (0.38281089082016573 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8613641509006986 = (66.70885672838702 - 9.248238994983579) / 66.70885672838702 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.031455260269219 < 36.52370628034422.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 189.62938554004668 (T) = (0 -8.315009314061852) / Math.Sqrt((2.98019386639231 / (299)) + (0.44198692203920215 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.79131736874117 = (39.84523898277241 - 8.315009314061852) / 39.84523898277241 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.83576646252926 < 36.61020315049516.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 95.52087035866855 (T) = (0 -8.581636569673396) / Math.Sqrt((5.1410608354046445 / (299)) + (2.288565221540868 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7858867454272725 = (40.0798941046337 - 8.581636569673396) / 40.0798941046337 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.991860529061652 < 36.45517354675939.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 224.75394258673495 (T) = (0 -8.210625926561356) / Math.Sqrt((3.7712181853489994 / (299)) + (0.17743461604814803 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7935185591995136 = (39.764474205189764 - 8.210625926561356) / 39.764474205189764 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.169271921553137 < 17.297198903424547.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 84.95121427553315 (T) = (0 -7.276480814530015) / Math.Sqrt((2.921958355013981 / (299)) + (0.2550619253853012 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6226451375742855 = (19.282859554943332 - 7.276480814530015) / 19.282859554943332 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddOperatorBenchmark - Duration of single invocation 64.56 ns 8.19 ns 0.13 0.16 True
XorBenchmark - Duration of single invocation 38.90 ns 9.57 ns 0.25 0.20 True
OneBenchmark - Duration of single invocation 29.37 ns 2.05 ns 0.07 0.21 True
MultiplyOperatorBenchmark - Duration of single invocation 64.50 ns 7.80 ns 0.12 0.16 True
BitwiseAndOperatorBenchmark - Duration of single invocation 45.00 ns 8.62 ns 0.19 0.22 True
AndNotBenchmark - Duration of single invocation 65.17 ns 9.57 ns 0.15 0.26 True
OnesComplementOperatorBenchmark - Duration of single invocation 29.08 ns 6.23 ns 0.21 0.22 True
BitwiseOrOperatorBenchmark - Duration of single invocation 38.47 ns 8.20 ns 0.21 0.16 True
ConditionalSelectBenchmark - Duration of single invocation 162.51 ns 15.01 ns 0.09 0.19 True
SubtractBenchmark - Duration of single invocation 64.38 ns 8.18 ns 0.13 0.22 True
BitwiseAndBenchmark - Duration of single invocation 38.48 ns 8.17 ns 0.21 0.22 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 41.11 ns 9.41 ns 0.23 0.20 True
OnesComplementBenchmark - Duration of single invocation 29.25 ns 6.20 ns 0.21 0.18 True
MultiplyBenchmark - Duration of single invocation 64.51 ns 10.26 ns 0.16 0.13 True
UnaryNegateOperatorBenchmark - Duration of single invocation 63.29 ns 6.18 ns 0.10 0.21 True
InequalityOperatorBenchmark - Duration of single invocation 31.54 ns 7.04 ns 0.22 0.35 True
NegateBenchmark - Duration of single invocation 63.32 ns 6.30 ns 0.10 0.18 True
AddBenchmark - Duration of single invocation 65.18 ns 8.23 ns 0.13 0.18 True
SubtractionOperatorBenchmark - Duration of single invocation 64.68 ns 8.18 ns 0.13 0.17 True
EqualityOperatorBenchmark - Duration of single invocation 18.21 ns 7.15 ns 0.39 0.30 True
BitwiseOrBenchmark - Duration of single invocation 38.68 ns 7.82 ns 0.20 0.21 True
EqualsAllBenchmark - Duration of single invocation 16.88 ns 6.90 ns 0.41 0.31 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.190135064148793 < 61.37894760300575.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 288.80451197631714 (T) = (0 -8.209752414344788) / Math.Sqrt((5.953370438328671 / (299)) + (0.4578767163237698 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8730619857960451 = (64.6752863263951 - 8.209752414344788) / 64.6752863263951 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.571125751216531 < 36.725636252199315.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 175.5354190929783 (T) = (0 -8.193795913803942) / Math.Sqrt((5.85624505229661 / (299)) + (0.3282651512753499 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7948690626573441 = (39.94422304090007 - 8.193795913803942) / 39.94422304090007 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.053599814441023 < 28.24652517642433.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 163.22545417797113 (T) = (0 -1.659399626838916) / Math.Sqrt((3.8503391088088903 / (299)) + (0.5309778394349735 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9478310852045718 = (31.808206732802073 - 1.659399626838916) / 31.808206732802073 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.800859071632353 < 61.486331464922024.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 360.29973319251616 (T) = (0 -7.976989643711362) / Math.Sqrt((5.430948291338373 / (299)) + (0.16072183360011297 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8762859836570225 = (64.47927146424885 - 7.976989643711362) / 64.47927146424885 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.61613505874068 < 36.86591025238227.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 152.49749053263577 (T) = (0 -8.585377591912854) / Math.Sqrt((4.160979401137787 / (299)) + (0.6740889049104023 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.7844998531294746 = (39.839312021773374 - 8.585377591912854) / 39.839312021773374 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.572559257432978 < 62.038622823409135.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 275.3922101731788 (T) = (0 -9.323395216561781) / Math.Sqrt((10.844845696116835 / (299)) + (0.1824679214618232 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.860442001808341 = (66.80659895793067 - 9.323395216561781) / 66.80659895793067 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.228846221865446 < 27.85895534135281.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 174.8765123606716 (T) = (0 -6.432568763977936) / Math.Sqrt((3.074139773282183 / (299)) + (0.20958626687075732 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7878726473901144 = (30.324089207900503 - 6.432568763977936) / 30.324089207900503 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.201663259739359 < 36.87080367242512.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 177.134421061835 (T) = (0 -8.363030484916747) / Math.Sqrt((4.021207138631352 / (299)) + (0.45517928011892433 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7902900468524837 = (39.8790346352037 - 8.363030484916747) / 39.8790346352037 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.00527401855796 < 153.77804859968418.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 376.0395690741354 (T) = (0 -14.602830569080483) / Math.Sqrt((34.26965685851177 / (299)) + (0.833610874132526 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9083012914281801 = (159.24794139977774 - 14.602830569080483) / 159.24794139977774 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.178881773166443 < 61.465605786531555.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 167.49746858635 (T) = (0 -8.345926361132829) / Math.Sqrt((25.728167574808754 / (299)) + (0.6619664759754795 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8706715345302586 = (64.53278735519751 - 8.345926361132829) / 64.53278735519751 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.173222753135102 < 36.40777797369166.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 220.27557078305338 (T) = (0 -8.12829810616273) / Math.Sqrt((4.324338279061713 / (299)) + (0.1548971151627668 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7957133570839864 = (39.78869097919652 - 8.12829810616273) / 39.78869097919652 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.412604239804788 < 36.48358387947238.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 186.4853091654345 (T) = (0 -8.259476304703396) / Math.Sqrt((5.73443800968655 / (299)) + (0.23946498688289944 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7929105920248183 = (39.883625075084666 - 8.259476304703396) / 39.883625075084666 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.197950366583465 < 27.746412294370465.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 153.4247183116712 (T) = (0 -6.338629888917646) / Math.Sqrt((2.7624358548460872 / (299)) + (0.37324313690583155 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7900451726334782 = (30.190446051769925 - 6.338629888917646) / 30.190446051769925 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.26001896288786 < 61.402852946180616.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 293.92916141697384 (T) = (0 -8.207980374416017) / Math.Sqrt((7.2345012550989445 / (299)) + (0.3092610951909818 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8725743765805559 = (64.41389223106243 - 8.207980374416017) / 64.41389223106243 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.178392295838233 < 60.10732067867396.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 289.48364873544983 (T) = (0 -6.445174004853538) / Math.Sqrt((8.95067630895026 / (299)) + (0.21907075629633166 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8983282561501186 = (63.39198838145097 - 6.445174004853538) / 63.39198838145097 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.040161205858384 < 18.405927747980783.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 65.03833276969121 (T) = (0 -7.4789696702938455) / Math.Sqrt((3.755036240150379 / (299)) + (0.5122681034829923 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6125409099342157 = (19.302604744733276 - 7.4789696702938455) / 19.302604744733276 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.30429591042761 < 60.170784815280314.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 272.69352421359173 (T) = (0 -6.424894046921568) / Math.Sqrt((7.173184726710894 / (299)) + (0.4832254441810622 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8983106514694185 = (63.181583319804396 - 6.424894046921568) / 63.181583319804396 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.228731106238499 < 61.45649196062457.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 223.05460644648102 (T) = (0 -8.187582133922703) / Math.Sqrt((15.537984489464554 / (299)) + (0.33329503888621254 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8743964522876717 = (65.18591459434606 - 8.187582133922703) / 65.18591459434606 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.178874748425612 < 61.34722292132478.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 341.78683121526416 (T) = (0 -8.12955330207118) / Math.Sqrt((6.279424005604554 / (299)) + (0.1425158258994457 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.8729362740208735 = (63.98012681767782 - 8.12955330207118) / 63.98012681767782 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.152059679796026 < 17.163427288735868.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 63.42059007076514 (T) = (0 -7.341075175811989) / Math.Sqrt((2.7765906265154325 / (299)) + (0.6293111754288196 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6159295097501738 = (19.113874567756667 - 7.341075175811989) / 19.113874567756667 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.824747690213392 < 36.50206883523937.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 201.03755060398368 (T) = (0 -8.209234818445022) / Math.Sqrt((5.949044600585179 / (299)) + (0.13475008033570862 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.7956766595288877 = (40.177665456706166 - 8.209234818445022) / 40.177665456706166 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int32>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.897170845798895 < 17.211234989556306.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 71.52970481101424 (T) = (0 -7.015462424309848) / Math.Sqrt((2.66942781316227 / (299)) + (0.4960171588026207 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.6336156763276161 = (19.14782366775873 - 7.015462424309848) / 19.14782366775873 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jun 6, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in BilinearTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Interpol_Vector - Duration of single invocation 1.44 ms 1.31 ms 0.92 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BilinearTest*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BilinearTest*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BilinearTest*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BilinearTest*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

BilinearTest.Interpol_Vector

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3143036736111111 < 1.3635640460289502.
IsChangePoint: Marked as a change because one of 3/29/2023 4:47:17 AM, 5/10/2023 10:36:48 PM, 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 27.889477183250715 (T) = (0 -1330182.1215985334) / Math.Sqrt((959291387.5407275 / (299)) + (453074318.661982 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.08828688296682628 = (1458991.9753782956 - 1330182.1215985334) / 1458991.9753782956 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
Compare 182591af3ea67a09bca7e0516b950dafce71d5a6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 119.90 ns 8.10 ns 0.07 0.15 True
AddOperatorBenchmark - Duration of single invocation 118.56 ns 8.14 ns 0.07 0.18 True
SubtractBenchmark - Duration of single invocation 122.75 ns 8.11 ns 0.07 0.15 True
AddBenchmark - Duration of single invocation 122.05 ns 8.80 ns 0.07 0.16 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.10086069606524 < 113.918836742738.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 330.63220532723176 (T) = (0 -8.287023677717452) / Math.Sqrt((29.89902782218184 / (299)) + (0.4931396358246373 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9324549253938973 = (122.68879301776239 - 8.287023677717452) / 122.68879301776239 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.137461464398767 < 113.09100496104048.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 359.6098466793392 (T) = (0 -8.173309061115157) / Math.Sqrt((27.463577441408198 / (299)) + (0.2506896346218591 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9336530765643087 = (123.19047572774633 - 8.173309061115157) / 123.19047572774633 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.110863852837648 < 116.30801931747824.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 460.75058900927615 (T) = (0 -8.350821959094704) / Math.Sqrt((10.079884396755595 / (299)) + (0.7647640499624532 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9332935982646053 = (125.18771424997608 - 8.350821959094704) / 125.18771424997608 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.803428243253611 < 119.23630249581345.
IsChangePoint: Marked as a change because one of 5/29/2023 10:47:10 AM, 6/5/2023 4:27:01 PM falls between 5/27/2023 6:38:16 PM and 6/5/2023 4:27:01 PM.
IsImprovementStdDev: Marked as improvement because 522.4170987052823 (T) = (0 -8.170911059973253) / Math.Sqrt((11.356683736732045 / (299)) + (0.318113838053612 / (25))) is greater than 1.9673585853226652 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (25) - 2, .975) and 0.9350533852278581 = (125.80965287628914 - 8.170911059973253) / 125.80965287628914 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

@BrzVlad the only change in the diff is dotnet/runtime#86859. Well done!

@kotlarmilos kotlarmilos mentioned this issue Jun 7, 2023
Closed
@kotlarmilos kotlarmilos mentioned this issue Jun 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos