Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 336 Regressions on 3/24/2023 7:13:05 PM #14570

Closed
performanceautofiler bot opened this issue Mar 28, 2023 · 24 comments
Closed

[Perf] Linux/arm64: 336 Regressions on 3/24/2023 7:13:05 PM #14570

performanceautofiler bot opened this issue Mar 28, 2023 · 24 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 1.28 ns 13.72 ns 10.69 0.09 False
NegateBenchmark - Duration of single invocation 0.92 ns 48.52 ns 52.52 0.06 False
SubtractionOperatorBenchmark - Duration of single invocation 1.09 ns 51.63 ns 47.45 0.08 False
OnesComplementBenchmark - Duration of single invocation 0.95 ns 11.09 ns 11.70 0.15 False
AddBenchmark - Duration of single invocation 1.15 ns 51.46 ns 44.71 0.05 False
SubtractBenchmark - Duration of single invocation 1.11 ns 52.21 ns 47.07 0.08 False
XorBenchmark - Duration of single invocation 1.12 ns 13.59 ns 12.10 0.09 False
BitwiseOrBenchmark - Duration of single invocation 1.14 ns 13.47 ns 11.79 0.09 False
AddOperatorBenchmark - Duration of single invocation 1.31 ns 51.20 ns 39.22 0.08 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.14 ns 13.24 ns 11.64 0.07 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.15 ns 13.53 ns 11.79 0.09 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.88 ns 11.09 ns 12.62 0.16 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.84 ns 48.59 ns 57.97 0.06 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.27 ns 13.35 ns 10.50 0.08 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.715611329258174 > 1.242007534140173.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -216.89080549673818 (T) = (0 -13.468303051251308) / Math.Sqrt((0.01179274942090533 / (27)) + (0.03066525588362944 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.68929218956177 = (1.152191495673121 - 13.468303051251308) / 1.152191495673121 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.5189170393852 > 0.9651314611038949.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -332.4296375355495 (T) = (0 -48.92119270167668) / Math.Sqrt((0.03672160108578582 / (27)) + (0.21393079233308981 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -49.54214634528552 = (0.9679286741695717 - 48.92119270167668) / 0.9679286741695717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.627691776960596 > 1.269204914098522.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -247.73561813230035 (T) = (0 -51.34716153813711) / Math.Sqrt((0.05322150018438026 / (27)) + (0.4301192586288372 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -44.043921541474894 = (1.1399354181642112 - 51.34716153813711) / 1.1399354181642112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.085824566015047 > 0.9817965978302264.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -289.3266836792468 (T) = (0 -11.150917485469796) / Math.Sqrt((0.02019658829874963 / (27)) + (0.00525217355353952 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.906078795423127 = (1.022449745196176 - 11.150917485469796) / 1.022449745196176 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.4638804454492 > 1.1985602854489708.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -532.8222728397416 (T) = (0 -51.54970230582933) / Math.Sqrt((0.00886130047776759 / (27)) + (0.09486528167443264 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -44.38476620207997 = (1.135837123767464 - 51.54970230582933) / 1.135837123767464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.2090755121765 > 1.2817146558113286.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -247.2851264938761 (T) = (0 -51.494126545873534) / Math.Sqrt((0.012838064418516818 / (27)) + (0.45091297788601487 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -44.24961188875967 = (1.138001507559119 - 51.494126545873534) / 1.138001507559119 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.588367473859083 > 1.2597434633172746.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -171.7948967271585 (T) = (0 -13.483684747618463) / Math.Sqrt((0.05609300911321309 / (27)) + (0.03417858165142694 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.107523496480198 = (1.1136616626462326 - 13.483684747618463) / 1.1136616626462326 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.474684932462031 > 1.3157966536475254.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -129.52966937814472 (T) = (0 -13.39562480465424) / Math.Sqrt((0.07719319906653094 / (27)) + (0.06739391097429409 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.991326624854283 = (1.117109492863724 - 13.39562480465424) / 1.117109492863724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.201474276193835 > 1.2567308427036223.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -242.0257263477622 (T) = (0 -51.31776885471336) / Math.Sqrt((0.023038485834078835 / (27)) + (0.46263409050210413 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -42.4066153880885 = (1.1822568609851993 - 51.31776885471336) / 1.1822568609851993 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.237973520098583 > 1.2131432074243347.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -195.0397528668844 (T) = (0 -13.382921510834484) / Math.Sqrt((0.055522431375430184 / (27)) + (0.020884839264610374 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.980113158641581 = (1.1170947497420771 - 13.382921510834484) / 1.1170947497420771 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.530577054291554 > 1.213840063946099.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -176.27158035701484 (T) = (0 -13.430381198992615) / Math.Sqrt((0.06882247300095148 / (27)) + (0.025593649344539372 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.969665894802267 = (1.1220347599530454 - 13.430381198992615) / 1.1220347599530454 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.086477283211865 > 0.9395284161650067.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -295.5072732554825 (T) = (0 -11.14093485909343) / Math.Sqrt((0.016007802406676125 / (27)) + (0.006415994741737423 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.069166276664921 = (1.0064836484189243 - 11.14093485909343) / 1.0064836484189243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.59238152863734 > 0.9600780600666644.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -337.76680232621703 (T) = (0 -49.068130100167814) / Math.Sqrt((0.04728309468492166 / (27)) + (0.20380628246052485 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -49.65963658239415 = (0.9685843288741744 - 49.068130100167814) / 0.9685843288741744 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.350765568872857 > 1.2841312424117144.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -156.39220910845697 (T) = (0 -13.424539241495212) / Math.Sqrt((0.09281574550902895 / (27)) + (0.029128716403701443 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.965454138325695 = (1.2242574791840761 - 13.424539241495212) / 1.2242574791840761 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 1.24 ns 17.33 ns 14.02 0.09 False
BitwiseAndBenchmark - Duration of single invocation 1.17 ns 13.66 ns 11.66 0.09 False
SubtractBenchmark - Duration of single invocation 1.24 ns 17.16 ns 13.82 0.08 False
XorBenchmark - Duration of single invocation 1.16 ns 13.47 ns 11.62 0.07 False
BitwiseOrBenchmark - Duration of single invocation 1.19 ns 13.54 ns 11.41 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.13 ns 13.45 ns 11.92 0.06 False
MaxBenchmark - Duration of single invocation 19.18 ns 20.49 ns 1.07 0.01 False
AddOperatorBenchmark - Duration of single invocation 1.27 ns 17.01 ns 13.40 0.06 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.95 ns 11.15 ns 11.69 0.14 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.90 ns 15.56 ns 17.25 0.10 False
OnesComplementBenchmark - Duration of single invocation 0.93 ns 11.15 ns 11.93 0.14 False
NegateBenchmark - Duration of single invocation 0.91 ns 15.49 ns 17.04 0.08 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.28 ns 13.43 ns 10.50 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 1.28 ns 17.03 ns 13.34 0.05 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.25 ns 13.44 ns 10.77 0.07 False
MinBenchmark - Duration of single invocation 19.12 ns 20.31 ns 1.06 0.03 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.AddBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.32839052096333 > 1.3344193561776607.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -242.83553261249548 (T) = (0 -17.26997434750032) / Math.Sqrt((0.05479421344413475 / (27)) + (0.026253870293846637 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.24824152391479 = (1.1325879328717816 - 17.26997434750032) / 1.1325879328717816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.658922556614268 > 1.2185574402444816.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -87.59817335905058 (T) = (0 -13.563261308958863) / Math.Sqrt((0.02642099972873989 / (27)) + (0.21153715546919716 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.214918139942313 = (1.1103849533471304 - 13.563261308958863) / 1.1103849533471304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.162749608275185 > 1.261281863281052.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -113.49465649610424 (T) = (0 -17.299641480976906) / Math.Sqrt((0.06675353481914864 / (27)) + (0.19269059127300767 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.8042091261683 = (1.253214966743904 - 17.299641480976906) / 1.253214966743904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.47476767821244 > 1.3196819624861957.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -257.3784298620389 (T) = (0 -13.459356956273355) / Math.Sqrt((0.016348935534790852 / (27)) + (0.01842498470999221 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.520283825306349 = (1.1683181734383565 - 13.459356956273355) / 1.1683181734383565 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.544569611575582 > 1.2339671905535956.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -77.3827477525371 (T) = (0 -13.607023604571344) / Math.Sqrt((0.013581540154425987 / (27)) + (0.27854693054123797 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.616228397860551 = (1.171380515131525 - 13.607023604571344) / 1.171380515131525 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.450721583919963 > 1.3400696873202829.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -165.53435101056618 (T) = (0 -13.435823270885361) / Math.Sqrt((0.06158482617243731 / (27)) + (0.03586632752013387 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.069064414832027 = (1.1132448058172333 - 13.435823270885361) / 1.1132448058172333 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.MaxBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.48913293330069 > 20.05826639206853.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -33.148761262559134 (T) = (0 -20.472767841418577) / Math.Sqrt((0.011522810972040385 / (16)) + (0.017386796839293645 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.06715467789607434 = (19.184442766798547 - 20.472767841418577) / 19.184442766798547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.012986144497198 > 1.3361167259298774.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -232.81546133368585 (T) = (0 -17.191545948320712) / Math.Sqrt((0.05825390382879366 / (27)) + (0.028718189820650754 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.418785089590623 = (1.1149740948089937 - 17.191545948320712) / 1.1149740948089937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.14716970749386 > 0.9875132288981626.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -166.54985998066232 (T) = (0 -11.183957402715496) / Math.Sqrt((0.04276445574158894 / (27)) + (0.023321253815634373 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.675036452710412 = (1.0476739308816008 - 11.183957402715496) / 1.0476739308816008 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.56248329927204 > 0.9503509812149556.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -189.76481865804956 (T) = (0 -15.674173961074972) / Math.Sqrt((0.0575648812893783 / (27)) + (0.04184105801675336 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.871852779446336 = (1.053948972836625 - 15.674173961074972) / 1.053948972836625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.147610629898061 > 0.9380869014414491.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -191.6748786721921 (T) = (0 -11.133969184009253) / Math.Sqrt((0.0615022585380967 / (27)) + (0.0054569846858279775 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.718342568948193 = (1.03877713484034 - 11.133969184009253) / 1.03877713484034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.491896373404343 > 0.9612540623601963.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -377.13497260560735 (T) = (0 -15.610550797507416) / Math.Sqrt((0.007449271575291965 / (27)) + (0.013503651070076969 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.814112973617622 = (0.9871278157396621 - 15.610550797507416) / 0.9871278157396621 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.43239085198979 > 1.236918767286962.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -248.71259223074594 (T) = (0 -13.471050035665984) / Math.Sqrt((0.012479065067538906 / (27)) + (0.021707340339238955 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.25700757658125 = (1.196681262229117 - 13.471050035665984) / 1.196681262229117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.033474851590057 > 1.2047378112177773.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -292.40267438009096 (T) = (0 -17.125519845114802) / Math.Sqrt((0.05970566422486905 / (27)) + (0.00838539823444961 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.506882904036791 = (1.1805099660899123 - 17.125519845114802) / 1.1805099660899123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.443143795354407 > 1.3027175323756366.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -238.34172612483496 (T) = (0 -13.396771703192085) / Math.Sqrt((0.013631000223055597 / (27)) + (0.023551490390793454 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.783765756326648 = (1.1368837416001265 - 13.396771703192085) / 1.1368837416001265 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.MinBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.30850579331452 > 20.107688014496087.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -17.299655132689363 (T) = (0 -20.53392997255517) / Math.Sqrt((0.0035794626106698414 / (16)) + (0.1266200761869335 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.06968544312383189 = (19.196232036765284 - 20.53392997255517) / 19.196232036765284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 1.15 ns 13.95 ns 12.09 0.08 False
BitwiseAndBenchmark - Duration of single invocation 1.25 ns 13.44 ns 10.75 0.06 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 0.09 ns 13.39 ns 149.99 0.08 False
SubtractionOperatorBenchmark - Duration of single invocation 1.15 ns 14.05 ns 12.20 0.07 False
OnesComplementBenchmark - Duration of single invocation 0.94 ns 11.20 ns 11.90 0.14 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.15 ns 13.59 ns 11.82 0.07 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.95 ns 12.59 ns 13.25 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.12 ns 13.51 ns 12.11 0.09 False
BitwiseOrBenchmark - Duration of single invocation 1.27 ns 13.66 ns 10.78 0.07 False
SubtractBenchmark - Duration of single invocation 1.24 ns 13.91 ns 11.17 0.06 False
NegateBenchmark - Duration of single invocation 0.88 ns 12.51 ns 14.18 0.11 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.95 ns 11.14 ns 11.75 0.14 False
XorBenchmark - Duration of single invocation 1.22 ns 13.41 ns 11.03 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.24 ns 13.96 ns 11.25 0.08 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.AddBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.954217579637238 > 1.2673676171070427.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -172.8229628516566 (T) = (0 -13.918909337630765) / Math.Sqrt((0.10523001552518743 / (27)) + (0.01671782387298971 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.610260392092997 = (1.198845578615105 - 13.918909337630765) / 1.198845578615105 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.435771020927563 > 1.1922514521787353.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -245.75945399701226 (T) = (0 -13.386125479939677) / Math.Sqrt((0.03437118485422311 / (27)) + (0.013045602456050753 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.160654285152605 = (1.1994032910550496 - 13.386125479939677) / 1.1994032910550496 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.389644883049685 > 1.2920658735859207.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -172.76530893362786 (T) = (0 -13.411446358655787) / Math.Sqrt((0.09771977777780917 / (27)) + (0.015363709761904997 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.407955012281098 = (1.1756223042795886 - 13.411446358655787) / 1.1756223042795886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.05326732771797 > 1.2613454534886372.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -214.32812594708463 (T) = (0 -13.954447793673857) / Math.Sqrt((0.018383545499331893 / (27)) + (0.031577625878940974 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.810181706263783 = (1.1815608041214825 - 13.954447793673857) / 1.1815608041214825 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.203488030565694 > 0.9699676000815451.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -53.23065562507008 (T) = (0 -11.317638637474058) / Math.Sqrt((0.03869999000028357 / (27)) + (0.4014553239976414 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.90409510785925 = (0.9507348971029302 - 11.317638637474058) / 0.9507348971029302 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.591463870057284 > 1.315766181084578.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -129.50556412572257 (T) = (0 -13.47215626856083) / Math.Sqrt((0.20481682873348722 / (27)) + (0.015231844329862803 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.168008358439113 = (1.206316814616331 - 13.47215626856083) / 1.206316814616331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.58675879811333 > 0.9348471009730513.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -45.023051384322066 (T) = (0 -12.970163523615748) / Math.Sqrt((0.020792748908324058 / (27)) + (0.770540908857094 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.118692406316574 = (0.9886780726233575 - 12.970163523615748) / 0.9886780726233575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.514244823555291 > 1.221319180567391.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -322.88633293606244 (T) = (0 -13.509146154842709) / Math.Sqrt((0.009569156719959627 / (27)) + (0.012179372837056042 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.597863006063463 = (1.1647961480300302 - 13.509146154842709) / 1.1647961480300302 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.65968638923586 > 1.2115233226787352.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -222.5810667594658 (T) = (0 -13.480805624948227) / Math.Sqrt((0.00610827752247539 / (27)) + (0.03115774327340189 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.514820712479354 = (1.1707351735262546 - 13.480805624948227) / 1.1707351735262546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.910547550807987 > 1.2652762417355892.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -411.0668080793775 (T) = (0 -13.872605379597404) / Math.Sqrt((0.022211604011733773 / (27)) + (0.001503684620164499 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.163886857352809 = (1.1404747135749385 - 13.872605379597404) / 1.1404747135749385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.511233546543862 > 0.9855582111968839.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -158.68304918277138 (T) = (0 -12.762820160289163) / Math.Sqrt((0.05320390375518317 / (27)) + (0.03907135789671119 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.149822426095637 = (0.9705697724831268 - 12.762820160289163) / 0.9705697724831268 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.135088603314232 > 0.9831342743232209.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -46.68895357853661 (T) = (0 -11.372633341271412) / Math.Sqrt((0.06382790370888425 / (27)) + (0.5127935431248347 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.940121542299316 = (1.039534460133721 - 11.372633341271412) / 1.039534460133721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.410625508813617 > 1.32072834599794.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -140.70657953031576 (T) = (0 -13.458263307349718) / Math.Sqrt((0.15740709850349052 / (27)) + (0.020369661973505752 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.952192816647617 = (1.1260078810479337 - 13.458263307349718) / 1.1260078810479337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.955210287166938 > 1.2463283533768026.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.723228979663595 (T) = (0 -14.13699099436951) / Math.Sqrt((0.05706052496318327 / (27)) + (0.6495921437756703 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.881298454527514 = (1.0974818295123536 - 14.13699099436951) / 1.0974818295123536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 0.90 ns 10.68 ns 11.88 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.11 ns 11.18 ns 10.11 0.08 False
ConditionalSelectBenchmark - Duration of single invocation 1.83 ns 32.87 ns 18.01 0.31 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.11 ns 9.58 ns 8.63 0.10 False
XorBenchmark - Duration of single invocation 1.32 ns 9.64 ns 7.31 0.35 False
SubtractBenchmark - Duration of single invocation 1.31 ns 10.78 ns 8.23 0.08 False
BitwiseAndBenchmark - Duration of single invocation 1.12 ns 9.16 ns 8.20 0.35 False
AndNotBenchmark - Duration of single invocation 1.30 ns 17.71 ns 13.62 0.08 False
OneBenchmark - Duration of single invocation 0.00 ns 8.62 ns ∞ 0.29 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.30 ns 11.66 ns 9.00 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.86 ns 8.98 ns 10.43 0.10 False
BitwiseOrBenchmark - Duration of single invocation 1.11 ns 9.53 ns 8.61 0.37 False
AddBenchmark - Duration of single invocation 1.08 ns 10.67 ns 9.89 0.08 False
NegateBenchmark - Duration of single invocation 0.92 ns 10.72 ns 11.70 0.07 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 0.15 ns 9.59 ns 62.18 0.09 False
OnesComplementBenchmark - Duration of single invocation 0.91 ns 9.08 ns 9.97 0.11 False
SubtractionOperatorBenchmark - Duration of single invocation 1.30 ns 10.82 ns 8.31 0.09 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.68055166289273 > 0.9592345918154853.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -167.08582036683697 (T) = (0 -10.687394417170829) / Math.Sqrt((0.0841896306367864 / (27)) + (0.0032390012180798997 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.532990790628283 = (0.9266802177502312 - 10.687394417170829) / 0.9266802177502312 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.182163926347213 > 1.2772040111274277.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.03247774424003 (T) = (0 -10.790557918054164) / Math.Sqrt((0.024555279988533428 / (27)) + (0.022990639654452187 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.9591878518011425 = (1.204412508873206 - 10.790557918054164) / 1.204412508873206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.87201280606714 > 1.9184388674270696.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -326.81357493491646 (T) = (0 -32.93267938382418) / Math.Sqrt((0.180004516211168 / (27)) + (0.027766708227470082 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.56978117344398 = (1.6010223495396712 - 32.93267938382418) / 1.6010223495396712 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.578275766625271 > 1.2037499570273178.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -146.1575153025039 (T) = (0 -9.555749517425872) / Math.Sqrt((0.08362766240535387 / (27)) + (0.0015903086415996568 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.74443355690076 = (1.2338861773707284 - 9.555749517425872) / 1.2338861773707284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.643511867926756 > 1.3768320815800732.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -191.27303396515055 (T) = (0 -9.596964996953892) / Math.Sqrt((0.047903157488417406 / (27)) + (0.0016882928217861835 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.003660360319105 = (1.1990719951753752 - 9.596964996953892) / 1.1990719951753752 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.779493864006588 > 1.301837970488459.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -104.84474594751902 (T) = (0 -10.845577006261808) / Math.Sqrt((0.08066438875174116 / (27)) + (0.061344698038069675 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.49000113896672 = (1.1428425400002304 - 10.845577006261808) / 1.1428425400002304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.158924496943905 > 1.234760658991739.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -159.43454666478803 (T) = (0 -9.560690948172628) / Math.Sqrt((0.025276613408789943 / (27)) + (0.019981959017163806 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.995558373223857 = (1.1957502530642772 - 9.560690948172628) / 1.1957502530642772 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.707574569168223 > 1.3908047361219305.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -330.4971445826757 (T) = (0 -17.741207567987775) / Math.Sqrt((0.06271192658058525 / (27)) + (0.0017995810087394544 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.060060777708566 = (1.261815851899834 - 17.741207567987775) / 1.261815851899834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.619338812107381 > 0.7681491888676496.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -85.38584179120679 (T) = (0 -8.75306590058938) / Math.Sqrt((0.07660205022236118 / (27)) + (0.06155697307733872 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.598989696109602 = (0.9118736635520021 - 8.75306590058938) / 0.9118736635520021 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.66312746666169 > 1.2710792032176026.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -37.39984300740208 (T) = (0 -9.957618698443982) / Math.Sqrt((0.05014182276846314 / (27)) + (0.577661224476081 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.05101306557519 = (1.236815617778727 - 9.957618698443982) / 1.236815617778727 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.983197865130498 > 0.9535485010058569.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -174.0394610054867 (T) = (0 -9.034010413546826) / Math.Sqrt((0.012894371648010106 / (27)) + (0.018451849488861503 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.462318741531329 = (0.9547353730429092 - 9.034010413546826) / 0.9547353730429092 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.526999112150268 > 1.1925775264576999.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -130.03383688160858 (T) = (0 -9.630054401727989) / Math.Sqrt((0.04440297433878309 / (27)) + (0.027767585417398395 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.802816499935968 = (1.2341767106540331 - 9.630054401727989) / 1.2341767106540331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.671039195289312 > 1.2946808244656374.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -140.24820783198527 (T) = (0 -10.755307018448768) / Math.Sqrt((0.0975509546787222 / (27)) + (0.010645026421446973 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.514620989371942 = (1.2631574596066788 - 10.755307018448768) / 1.2631574596066788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.721109412249145 > 0.9649024896837668.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -104.5353616631911 (T) = (0 -10.846971169642948) / Math.Sqrt((0.016005865579326487 / (27)) + (0.0914802749876736 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.068000208220166 = (0.9800299029256377 - 10.846971169642948) / 0.9800299029256377 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.587358005916114 > 1.2826598906115043.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -94.7607271447345 (T) = (0 -9.512250784081305) / Math.Sqrt((0.12223227519636894 / (27)) + (0.03673224609151229 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.587740780280651 = (1.1076546238940435 - 9.512250784081305) / 1.1076546238940435 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.080720011594876 > 0.9802134828920795.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -44.21226939301067 (T) = (0 -9.200055688727026) / Math.Sqrt((0.008484377307737045 / (27)) + (0.37695354986643326 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.405571187294173 = (0.9781495993731063 - 9.200055688727026) / 0.9781495993731063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.815226192504213 > 1.3220760451040994.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -84.59844338275846 (T) = (0 -10.85341630136248) / Math.Sqrt((0.03405586964874793 / (27)) + (0.1289990696760387 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.95517956199461 = (1.2119708182540418 - 10.85341630136248) / 1.2119708182540418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 1.30 ns 13.29 ns 10.25 0.07 False
SubtractionOperatorBenchmark - Duration of single invocation 1.25 ns 36.47 ns 29.24 0.06 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.30 ns 13.43 ns 10.34 0.09 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.88 ns 11.38 ns 12.86 0.13 False
BitwiseOrBenchmark - Duration of single invocation 1.20 ns 13.23 ns 11.04 0.07 False
AddBenchmark - Duration of single invocation 1.23 ns 36.60 ns 29.64 0.05 False
AddOperatorBenchmark - Duration of single invocation 1.13 ns 36.38 ns 32.22 0.07 False
OnesComplementBenchmark - Duration of single invocation 0.97 ns 11.20 ns 11.57 0.14 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.12 ns 13.46 ns 12.03 0.07 False
SubtractBenchmark - Duration of single invocation 1.28 ns 36.41 ns 28.54 0.04 False
AbsBenchmark - Duration of single invocation 7.83 ns 10.08 ns 1.29 0.07 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.15 ns 13.36 ns 11.64 0.08 False
XorBenchmark - Duration of single invocation 1.23 ns 13.21 ns 10.77 0.09 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.89 ns 32.70 ns 36.64 0.04 False
NegateBenchmark - Duration of single invocation 0.91 ns 32.83 ns 35.92 0.06 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.287791208478408 > 1.3316529094210263.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -157.35972938108023 (T) = (0 -13.435741453955115) / Math.Sqrt((0.09442369647043643 / (27)) + (0.028228057957840468 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.361965588351952 = (1.1825191116340958 - 13.435741453955115) / 1.1825191116340958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.469254515670364 > 1.3179874173488673.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -119.64343061281109 (T) = (0 -35.957756159198226) / Math.Sqrt((0.11032763131131315 / (27)) + (0.882745325464618 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -28.654475996223947 = (1.2125574622791144 - 35.957756159198226) / 1.2125574622791144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.43222602707897 > 1.2509756905392542.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -38.637788155098214 (T) = (0 -13.73706171914933) / Math.Sqrt((0.09528829949613978 / (27)) + (1.1393295347534578 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.57817831024392 = (1.0921344395286232 - 13.73706171914933) / 1.0921344395286232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.37505866548387 > 0.9421397993397489.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -130.71847753139863 (T) = (0 -11.159548579333658) / Math.Sqrt((0.010369761463643278 / (27)) + (0.06250675506367671 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.408436087570623 = (0.9781839065121182 - 11.159548579333658) / 0.9781839065121182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.225076156508925 > 1.347430431560198.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -139.578809806786 (T) = (0 -13.368428093311623) / Math.Sqrt((0.04951627953589243 / (27)) + (0.0634771606837378 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.172404408607958 = (1.1965578405854789 - 13.368428093311623) / 1.1965578405854789 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.60326395636149 > 1.332761631689841.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -176.1147483453808 (T) = (0 -36.20536887124299) / Math.Sqrt((0.009614398759615915 / (27)) + (0.43138925041120224 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.923714749722947 = (1.1707962372653615 - 36.20536887124299) / 1.1707962372653615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.38194670881759 > 1.213838208924319.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -154.9607548345735 (T) = (0 -35.89481233949651) / Math.Sqrt((0.012817339305648556 / (27)) + (0.5471478465217029 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.680180202649176 = (1.1699674546369536 - 35.89481233949651) / 1.1699674546369536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.202360360817107 > 1.0057342778442204.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -195.02060576045514 (T) = (0 -11.169499860317565) / Math.Sqrt((0.049406070180206214 / (27)) + (0.009707185239184218 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.028034795460195 = (1.0128277673657327 - 11.169499860317565) / 1.0128277673657327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.464770292125158 > 1.2989441462981572.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -218.80184056333482 (T) = (0 -13.481662943745341) / Math.Sqrt((0.031034822425729254 / (27)) + (0.0224278898846272 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.963090746332174 = (1.1269381157105036 - 13.481662943745341) / 1.1269381157105036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.405513387268016 > 1.328453547363778.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -142.53580815885238 (T) = (0 -36.10039670923292) / Math.Sqrt((0.013324998945398615 / (27)) + (0.6548204349134588 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.598545146657077 = (1.179807619486671 - 36.10039670923292) / 1.179807619486671 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AbsBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.07534409611291 > 8.262633725726708.
IsChangePoint: Marked as a change because one of 12/12/2022 5:25:39 PM, 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -33.7131441015279 (T) = (0 -9.931283557140942) / Math.Sqrt((0.0029709893902455957 / (16)) + (0.08006901048960777 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2657460430729452 = (7.846189693020909 - 9.931283557140942) / 7.846189693020909 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.361839362553516 > 1.2043604460802815.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -143.3876195373268 (T) = (0 -13.338000121184052) / Math.Sqrt((0.11363636291720011 / (27)) + (0.032330697892417444 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.975518218077088 = (1.2152501463863337 - 13.338000121184052) / 1.2152501463863337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.20543280649677 > 1.2907285642033919.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -211.0379580778871 (T) = (0 -13.392009452087906) / Math.Sqrt((0.012274027086417541 / (27)) + (0.03186439470150808 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.399091385691394 = (1.1748313088267812 - 13.392009452087906) / 1.1748313088267812 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.69664646520911 > 0.953274121902396.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -700.9582070952117 (T) = (0 -32.87046590644848) / Math.Sqrt((0.01184882447269539 / (27)) + (0.0179367131383981 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -32.43937241394266 = (0.9829869262959919 - 32.87046590644848) / 0.9829869262959919 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.82796040553179 > 0.9522485303689922.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -527.2989726316156 (T) = (0 -32.80488420451573) / Math.Sqrt((0.06152544816295862 / (27)) + (0.015041736141423465 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -33.0018346514564 = (0.964797474630111 - 32.80488420451573) / 0.964797474630111 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 1.20 ns 9.59 ns 7.97 0.36 False
OnesComplementBenchmark - Duration of single invocation 0.94 ns 8.89 ns 9.46 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.30 ns 9.61 ns 7.40 0.08 False
ConditionalSelectBenchmark - Duration of single invocation 1.94 ns 32.92 ns 16.99 0.32 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.19 ns 11.77 ns 9.92 0.10 False
OneBenchmark - Duration of single invocation 0.79 ns 6.45 ns 8.12 0.24 False
BitwiseOrBenchmark - Duration of single invocation 1.11 ns 9.52 ns 8.61 0.33 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.15 ns 9.55 ns 8.33 0.08 False
BitwiseAndBenchmark - Duration of single invocation 1.30 ns 9.55 ns 7.36 0.38 False
AndNotBenchmark - Duration of single invocation 1.35 ns 17.86 ns 13.19 0.05 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.85 ns 9.24 ns 10.83 0.13 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.XorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.586659520961126 > 1.307893885760905.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -166.3360287995319 (T) = (0 -9.648919498882476) / Math.Sqrt((0.028558880190174836 / (27)) + (0.016623456471874767 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.920677936420319 = (1.218193641546196 - 9.648919498882476) / 1.218193641546196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.894943936516553 > 0.9758077266802266.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -264.1971565056461 (T) = (0 -9.04271214943855) / Math.Sqrt((0.01105398853028745 / (27)) + (0.005760218791737967 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.298192415071 = (0.9725236632855271 - 9.04271214943855) / 0.9725236632855271 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.610355804595713 > 1.2714734720280532.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.35441037268024 (T) = (0 -9.775373966658027) / Math.Sqrt((0.0847556271435929 / (27)) + (0.25947958410830807 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.038401831815548 = (1.2160842629150037 - 9.775373966658027) / 1.2160842629150037 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.917342778751156 > 1.771479971142576.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -325.0386229449861 (T) = (0 -32.82782326270633) / Math.Sqrt((0.19456915205644812 / (27)) + (0.02271430842020051 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.444839183545177 = (1.5308029583124794 - 32.82782326270633) / 1.5308029583124794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.767468683690241 > 1.2388140307318716.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -42.567897219925484 (T) = (0 -9.79141796243532) / Math.Sqrt((0.042549958927619175 / (27)) + (0.43015418605432326 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.121019177903683 = (1.2056883191553827 - 9.79141796243532) / 1.2056883191553827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.452020858262886 > 1.0405260512709835.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -35.13018455757754 (T) = (0 -6.197314824199761) / Math.Sqrt((0.045800709178437586 / (27)) + (0.22113483146677249 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -5.133125420744468 = (1.0104660183922183 - 6.197314824199761) / 1.0104660183922183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.51929150226407 > 1.2206162381932706.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -166.04407642094628 (T) = (0 -9.588614807770991) / Math.Sqrt((0.056322717465542925 / (27)) + (0.004982000328021888 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.846590480850195 = (1.2220103535634044 - 9.588614807770991) / 1.2220103535634044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.554647885604426 > 1.2671733992782996.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -204.63944766329908 (T) = (0 -9.590166737189033) / Math.Sqrt((0.04052860986856959 / (27)) + (0.002060843453752701 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.117081160176497 = (1.181479715176398 - 9.590166737189033) / 1.181479715176398 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.546630752638425 > 1.287674259297134.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -163.0628208265329 (T) = (0 -9.594105950057129) / Math.Sqrt((0.06547688633412282 / (27)) + (0.0022115439840168652 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.750674312753216 = (1.237841452616667 - 9.594105950057129) / 1.237841452616667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.86494256450413 > 1.3615631324007036.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -371.38603552576143 (T) = (0 -17.764704206368194) / Math.Sqrt((0.04641514716448779 / (27)) + (0.0029005878218330327 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.472073977837775 = (1.2275161275137678 - 17.764704206368194) / 1.2275161275137678 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.237998634538757 > 0.9612260226053142.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -44.832462395582596 (T) = (0 -9.271245603616592) / Math.Sqrt((0.02389882907294542 / (27)) + (0.36365854615576376 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.16823288321588 = (1.0112358315624046 - 9.271245603616592) / 1.0112358315624046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrBenchmark - Duration of single invocation 1.25 ns 13.57 ns 10.87 0.09 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.88 ns 12.48 ns 14.24 0.13 False
NegateBenchmark - Duration of single invocation 0.91 ns 12.87 ns 14.18 0.13 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.12 ns 13.48 ns 12.03 0.07 False
XorBenchmark - Duration of single invocation 1.30 ns 13.52 ns 10.37 0.07 False
BitwiseAndBenchmark - Duration of single invocation 1.10 ns 13.45 ns 12.23 0.08 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.15 ns 13.22 ns 11.49 0.08 False
SubtractionOperatorBenchmark - Duration of single invocation 0.00 ns 13.83 ns ∞ 0.10 False
AddOperatorBenchmark - Duration of single invocation 1.28 ns 13.79 ns 10.79 0.09 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.15 ns 13.36 ns 11.62 0.07 False
OnesComplementBenchmark - Duration of single invocation 0.92 ns 11.08 ns 11.98 0.13 False
AbsBenchmark - Duration of single invocation 7.85 ns 9.83 ns 1.25 0.06 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.91 ns 13.41 ns 14.80 0.14 False
SubtractBenchmark - Duration of single invocation 1.10 ns 13.91 ns 12.63 0.11 False
AddBenchmark - Duration of single invocation 1.15 ns 13.80 ns 12.04 0.10 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.565950091203746 > 1.3152807332068672.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -193.38214180775145 (T) = (0 -13.453189557884098) / Math.Sqrt((0.07699433916006307 / (27)) + (0.013457944755252743 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.137912141109922 = (1.1083610922111933 - 13.453189557884098) / 1.1083610922111933 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.484376042985344 > 0.9453293702692707.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -161.6046544870495 (T) = (0 -12.683313915955164) / Math.Sqrt((0.02456005296040716 / (27)) + (0.04744485394425735 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.628803679762365 = (1.0043163420364314 - 12.683313915955164) / 1.0043163420364314 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.8650594309393 > 0.9702380551698953.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -179.27897255976015 (T) = (0 -12.711960646773962) / Math.Sqrt((0.014645373251700462 / (27)) + (0.04091796821212168 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.616224395384757 = (1.0075883440551539 - 12.711960646773962) / 1.0075883440551539 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.477549713454184 > 1.2064033434889998.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -182.00285239639254 (T) = (0 -13.458554534351798) / Math.Sqrt((0.016670972068809096 / (27)) + (0.04377044540529117 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.026046880113297 = (1.1191170854827905 - 13.458554534351798) / 1.1191170854827905 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.518098489525954 > 1.3435501576682385.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -210.19643344979374 (T) = (0 -13.454229861639135) / Math.Sqrt((0.01653256725063135 / (27)) + (0.030925774537383422 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.648264916551385 = (1.155041541210279 - 13.454229861639135) / 1.155041541210279 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.453350857306491 > 1.2539127545524822.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -178.92911228013952 (T) = (0 -13.395649438572105) / Math.Sqrt((0.014231059692429566 / (27)) + (0.0456418373329415 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.54986809546035 = (1.1598097335706574 - 13.395649438572105) / 1.1598097335706574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.22107995370579 > 1.284732765908938.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -189.4694926950671 (T) = (0 -13.432505218320237) / Math.Sqrt((0.05507081569485842 / (27)) + (0.02401755949861324 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.995502976497503 = (1.119795080259512 - 13.432505218320237) / 1.119795080259512 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.830218938997897 > 1.223076975888571.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -204.02631207539204 (T) = (0 -13.925165493808954) / Math.Sqrt((0.06019948760507145 / (27)) + (0.018816855822462376 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.453506955598177 = (1.1181722179509626 - 13.925165493808954) / 1.1181722179509626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.794103701216432 > 1.261944583840809.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -198.49784137173754 (T) = (0 -13.876805465061524) / Math.Sqrt((0.10227667703692485 / (27)) + (0.0041475873191458784 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.014380474887886 = (1.0662670798535316 - 13.876805465061524) / 1.0662670798535316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.355883187407953 > 1.338873155018289.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -267.10549250654105 (T) = (0 -13.447524651548274) / Math.Sqrt((0.026614174631561214 / (27)) + (0.012522717801360161 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.82652893728428 = (1.1370643679865906 - 13.447524651548274) / 1.1370643679865906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.076414862709292 > 0.9491749531765663.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -194.78301871122983 (T) = (0 -11.17219520142504) / Math.Sqrt((0.04639731477558411 / (27)) + (0.011252700192256779 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.474113056415474 = (0.9736870419956666 - 11.17219520142504) / 0.9736870419956666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AbsBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.833276592157885 > 8.164538562447303.
IsChangePoint: Marked as a change because one of 12/15/2022 1:21:30 AM, 12/30/2022 6:06:02 AM, 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -61.40849570416568 (T) = (0 -9.835293877086166) / Math.Sqrt((0.0058685594736244875 / (16)) + (0.016023936513372056 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.260433622620138 = (7.803103392815687 - 9.835293877086166) / 7.803103392815687 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.4094951502971 > 0.9498609683191308.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -49.420494965721154 (T) = (0 -11.362401046752634) / Math.Sqrt((0.027936221853505125 / (27)) + (0.47252085200514565 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.397537709698128 = (0.9969171707222695 - 11.362401046752634) / 0.9969171707222695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.906449710129625 > 1.193631207372802.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -246.69054692821686 (T) = (0 -13.922049013061702) / Math.Sqrt((0.041924662868266196 / (27)) + (0.012508324367215523 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.34592097663376 = (1.127663868852795 - 13.922049013061702) / 1.127663868852795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.804854175975516 > 1.2040046807371088.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -283.0512635212445 (T) = (0 -13.950503892629223) / Math.Sqrt((0.01659951772320537 / (27)) + (0.015698095646053966 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.02455999849455 = (1.160167515017248 - 13.950503892629223) / 1.160167515017248 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 1.11 ns 13.49 ns 12.11 0.08 False
SubtractionOperatorBenchmark - Duration of single invocation 1.31 ns 49.41 ns 37.65 0.07 False
XorBenchmark - Duration of single invocation 1.13 ns 13.02 ns 11.51 0.09 False
AddOperatorBenchmark - Duration of single invocation 1.20 ns 50.60 ns 42.15 0.08 False
BitwiseOrBenchmark - Duration of single invocation 1.17 ns 13.45 ns 11.46 0.08 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.12 ns 13.23 ns 11.84 0.06 False
AbsBenchmark - Duration of single invocation 7.99 ns 9.83 ns 1.23 0.09 False
OnesComplementBenchmark - Duration of single invocation 0.90 ns 11.09 ns 12.37 0.16 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.13 ns 13.24 ns 11.71 0.08 False
AddBenchmark - Duration of single invocation 1.31 ns 52.04 ns 39.73 0.07 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.91 ns 48.82 ns 53.77 0.06 False
BitwiseAndOperatorBenchmark - Duration of single invocation 0.05 ns 13.30 ns 243.77 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.91 ns 11.17 ns 12.29 0.12 False
SubtractBenchmark - Duration of single invocation 1.18 ns 52.04 ns 43.94 0.06 False
NegateBenchmark - Duration of single invocation 0.88 ns 48.71 ns 55.31 0.06 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.489490377447488 > 1.2114211730322155.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -205.99573972918895 (T) = (0 -13.366606206903338) / Math.Sqrt((0.01592977867761047 / (27)) + (0.03229542439346247 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.780304486385992 = (1.1346571068982614 - 13.366606206903338) / 1.1346571068982614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.41246929874478 > 1.3025351318389626.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.34871021798833 (T) = (0 -51.45363246165723) / Math.Sqrt((0.0320256478396485 / (27)) + (0.8896541402361742 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -41.11028025994374 = (1.221878176636148 - 51.45363246165723) / 1.221878176636148 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.017129697306956 > 1.3049292802201302.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -203.82425208724484 (T) = (0 -13.436794920918055) / Math.Sqrt((0.010730319027395886 / (27)) + (0.03557756646367098 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.648305271697742 = (1.153540760437131 - 13.436794920918055) / 1.153540760437131 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.59748460761231 > 1.2950017656898507.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -184.2966828413497 (T) = (0 -51.471582607509916) / Math.Sqrt((0.032913623625945514 / (27)) + (0.8072640632241971 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -44.45434040290407 = (1.1323799256851916 - 51.471582607509916) / 1.1323799256851916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.44547746543381 > 1.2852516830856024.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -174.39144450714122 (T) = (0 -13.460939539057419) / Math.Sqrt((0.027600863287543645 / (27)) + (0.04355670070863335 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.686109700532798 = (1.1518751649613304 - 13.460939539057419) / 1.1518751649613304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.227306501049439 > 1.1784351937829018.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -82.66347228091715 (T) = (0 -13.499236602931147) / Math.Sqrt((0.01186794176688604 / (27)) + (0.2401583199104158 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.610603238516184 = (1.1626645339278967 - 13.499236602931147) / 1.1626645339278967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AbsBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.827295374453241 > 8.250370167116234.
IsChangePoint: Marked as a change because one of 12/30/2022 6:06:02 AM, 3/6/2023 9:06:14 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -60.4575189761328 (T) = (0 -9.854072463558536) / Math.Sqrt((0.010555714552462377 / (16)) + (0.009771457600088315 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2560348071912263 = (7.845381678231066 - 9.854072463558536) / 7.845381678231066 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.089074485211947 > 0.9336889368800567.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -269.8692187919195 (T) = (0 -11.133089349384106) / Math.Sqrt((0.034911129256211544 / (27)) + (0.001398808095711865 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.560453086582905 = (0.9630322675073351 - 11.133089349384106) / 0.9630322675073351 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.236356438383456 > 1.188243489159241.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -141.80929056755477 (T) = (0 -13.383118246871135) / Math.Sqrt((0.0804387111475114 / (27)) + (0.048174880656293675 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.985321116677408 = (1.2182728301454475 - 13.383118246871135) / 1.2182728301454475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.04184291148046 > 1.217524493127895.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -337.8692873784897 (T) = (0 -51.622263323769154) / Math.Sqrt((0.14227050299789215 / (27)) + (0.18796737411100378 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -45.804732991659904 = (1.1029282729370058 - 51.622263323769154) / 1.1029282729370058 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.819687117805294 > 0.973107056116354.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -232.14050212586204 (T) = (0 -48.974906323242706) / Math.Sqrt((0.06488757047454381 / (27)) + (0.4435933367924072 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -48.537378436668526 = (0.9886455010100117 - 48.974906323242706) / 0.9886455010100117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.29748974668866 > 1.2476457705183248.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -92.3968009322368 (T) = (0 -13.513305559247176) / Math.Sqrt((0.141459206551949 / (27)) + (0.142188874353758 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.746975002484369 = (1.0601186208189355 - 13.513305559247176) / 1.0601186208189355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.166151720070596 > 0.9657173218367224.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -210.74084485077262 (T) = (0 -11.101348347766669) / Math.Sqrt((0.027438137489714035 / (27)) + (0.014136986720402565 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.196749548523309 = (0.9914795628549875 - 11.101348347766669) / 0.9914795628549875 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.040160284573794 > 1.2844884839998525.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -373.81613574746876 (T) = (0 -51.759714849848905) / Math.Sqrt((0.04853798520031152 / (27)) + (0.18144977071332438 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -42.12271870289389 = (1.2002887667278594 - 51.759714849848905) / 1.2002887667278594 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.714990874234395 > 0.9311717143338746.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -342.0831705300125 (T) = (0 -49.02409011871588) / Math.Sqrt((0.05811609519891752 / (27)) + (0.19319305980113632 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -48.439332665391156 = (0.9916009678066315 - 49.02409011871588) / 0.9916009678066315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.58 ns 9.46 ns 6.00 0.19 False
BitwiseAndBenchmark - Duration of single invocation 1.49 ns 9.51 ns 6.38 0.11 False
OnesComplementBenchmark - Duration of single invocation 0.87 ns 9.03 ns 10.38 0.10 False
ConditionalSelectBenchmark - Duration of single invocation 2.20 ns 32.78 ns 14.89 0.07 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.68 ns 9.44 ns 5.63 0.11 False
AndNotBenchmark - Duration of single invocation 1.23 ns 17.71 ns 14.38 0.09 False
XorBenchmark - Duration of single invocation 1.60 ns 9.57 ns 5.99 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.71 ns 9.44 ns 5.52 0.15 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.90 ns 9.08 ns 10.11 0.12 False
OneBenchmark - Duration of single invocation 0.68 ns 5.76 ns 8.50 0.28 False
BitwiseOrBenchmark - Duration of single invocation 1.50 ns 9.45 ns 6.30 0.14 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.458130326178322 > 1.181917373499217.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -38.85719488768469 (T) = (0 -9.724355237637997) / Math.Sqrt((0.12226877372535484 / (27)) + (0.4791415878079247 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.080219776832379 = (1.2034765769019904 - 9.724355237637997) / 1.2034765769019904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.505228473464316 > 1.5713169610903805.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -40.940391542610534 (T) = (0 -9.749696261743605) / Math.Sqrt((0.04809844925999861 / (27)) + (0.4544624592586105 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.795856059748035 = (1.250625484490887 - 9.749696261743605) / 1.250625484490887 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.031345728443476 > 0.9800223841175306.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -195.62964349481356 (T) = (0 -9.05888757033394) / Math.Sqrt((0.02298456146324766 / (27)) + (0.00930441530324628 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.06189564107796 = (0.9996680528155296 - 9.05888757033394) / 0.9996680528155296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.77708281693457 > 2.108433684744829.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -279.17503992180474 (T) = (0 -32.77328724977803) / Math.Sqrt((0.30195480412455283 / (27)) + (0.013479642951449145 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -18.57228498250577 = (1.6744742516814806 - 32.77328724977803) / 1.6744742516814806 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.4397793251836 > 1.5439741088371357.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -128.77586490168162 (T) = (0 -9.57524768582141) / Math.Sqrt((0.09591200403723452 / (27)) + (0.006754682977634976 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.580699563406759 = (1.2631087151959763 - 9.57524768582141) / 1.2631087151959763 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.70689141452008 > 1.4648808534036566.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -82.79995007909442 (T) = (0 -17.911575764973946) / Math.Sqrt((0.10031734547041923 / (27)) + (0.4071909769429928 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.919091869452417 = (1.2005808343903819 - 17.911575764973946) / 1.2005808343903819 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.56537313613275 > 1.413634415974096.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -105.09908635508201 (T) = (0 -9.532856911311482) / Math.Sqrt((0.1320436383163384 / (27)) + (0.012814660429705837 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.038597045568081 = (1.3543688961870513 - 9.532856911311482) / 1.3543688961870513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.444749929324946 > 1.4752601360518052.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.53520181401132 (T) = (0 -9.723045498242385) / Math.Sqrt((0.13791458149824368 / (27)) + (0.40099713005698795 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.745349603357753 = (1.2553397840204998 - 9.723045498242385) / 1.2553397840204998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.07519085666279 > 0.9578180581182252.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -162.5948095843051 (T) = (0 -9.040696594487644) / Math.Sqrt((0.050415039231808886 / (27)) + (0.0067366523295964795 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.575819266679312 = (0.9441172961509706 - 9.040696594487644) / 0.9441172961509706 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.76315357444301 > 1.0894400784866052.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.500823548004604 (T) = (0 -5.908510077609628) / Math.Sqrt((0.044511819943562354 / (27)) + (0.13223718232028797 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -4.593811424255668 = (1.056258359370031 - 5.908510077609628) / 1.056258359370031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.449366694392989 > 1.5718990425987132.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -128.97363120322512 (T) = (0 -9.550984819143668) / Math.Sqrt((0.06898819240216157 / (27)) + (0.01728241727471331 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.54272443754219 = (1.266251325794937 - 9.550984819143668) / 1.266251325794937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 1.32 ns 10.67 ns 8.08 0.11 False
BitwiseOrBenchmark - Duration of single invocation 1.13 ns 9.60 ns 8.46 0.34 False
XorBenchmark - Duration of single invocation 1.59 ns 9.50 ns 5.97 0.38 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.92 ns 8.94 ns 9.73 0.09 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.51 ns 9.44 ns 6.27 0.13 False
OneBenchmark - Duration of single invocation 0.81 ns 9.35 ns 11.55 0.30 False
NegateBenchmark - Duration of single invocation 1.35 ns 10.77 ns 8.00 0.08 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.73 ns 9.50 ns 5.50 0.10 False
BitwiseAndBenchmark - Duration of single invocation 1.10 ns 9.64 ns 8.79 0.36 False
AndNotBenchmark - Duration of single invocation 1.25 ns 17.71 ns 14.16 0.09 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.93 ns 10.88 ns 11.64 0.09 False
AddOperatorBenchmark - Duration of single invocation 1.34 ns 10.74 ns 8.02 0.06 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.71 ns 9.61 ns 5.61 0.11 False
SubtractBenchmark - Duration of single invocation 1.12 ns 10.81 ns 9.69 0.08 False
AddBenchmark - Duration of single invocation 1.48 ns 10.67 ns 7.21 0.08 False
ConditionalSelectBenchmark - Duration of single invocation 1.87 ns 32.95 ns 17.64 0.29 False
OnesComplementBenchmark - Duration of single invocation 0.93 ns 9.08 ns 9.72 0.11 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.674498575683938 > 1.306831530544979.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -221.35646613897015 (T) = (0 -10.755575055106023) / Math.Sqrt((0.03753499994414299 / (27)) + (0.005030837274490238 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.666714459157296 = (1.2410210473406824 - 10.755575055106023) / 1.2410210473406824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.60263005001879 > 1.459070120838497.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.00901187904724 (T) = (0 -9.747940632102875) / Math.Sqrt((0.0402122871906064 / (27)) + (0.46479141084327685 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.324700091927824 = (1.1709659836941295 - 9.747940632102875) / 1.1709659836941295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.502319043738595 > 1.2237373243591474.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -39.58699981900448 (T) = (0 -9.744637647103042) / Math.Sqrt((0.042215744232202417 / (27)) + (0.4904831746870223 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.858131056516264 = (1.2400706449178416 - 9.744637647103042) / 1.2400706449178416 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.938678034605635 > 1.1969887369244225.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -190.176781927524 (T) = (0 -9.049145724974023) / Math.Sqrt((0.03450067381812851 / (27)) + (0.00537717123173566 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.571262751964833 = (1.055754092114332 - 9.049145724974023) / 1.055754092114332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.44462543016382 > 1.706489394264726.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -40.91284666456528 (T) = (0 -9.734601562380762) / Math.Sqrt((0.06307486784080095 / (27)) + (0.4383507583005848 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.311552193483578 = (1.3314001329371248 - 9.734601562380762) / 1.3314001329371248 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.354809630595613 > 0.8459339787076425.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -82.54820960551926 (T) = (0 -8.838996415619143) / Math.Sqrt((0.05664448974881953 / (27)) + (0.07712345274056195 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.203129329559054 = (0.9604337936694671 - 8.838996415619143) / 0.9604337936694671 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.773367081799417 > 0.9710137413035501.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -203.71023022644397 (T) = (0 -10.789064552696152) / Math.Sqrt((0.0260478560680618 / (27)) + (0.014713189656785853 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.633962293016848 = (1.0145855566726203 - 10.789064552696152) / 1.0145855566726203 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.498959646982826 > 1.3569633011043984.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -111.44826533702 (T) = (0 -9.583525161552434) / Math.Sqrt((0.08511594087039377 / (27)) + (0.025793818574195427 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.258578038301419 = (1.320303386005212 - 9.583525161552434) / 1.320303386005212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.64016610129293 > 1.1954251596663716.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -189.12846676635783 (T) = (0 -9.571540468244937) / Math.Sqrt((0.042104591160871986 / (27)) + (0.004241701558337444 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.778672715069549 = (1.2304850478799656 - 9.571540468244937) / 1.2304850478799656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.709944727974417 > 1.7352211086379818.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -180.90407630435624 (T) = (0 -17.727689027186866) / Math.Sqrt((0.10375501132303773 / (27)) + (0.047263134820607046 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.601917028016901 = (1.406745417207098 - 17.727689027186866) / 1.406745417207098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.878303055799798 > 0.9635791265308488.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.15363728370306 (T) = (0 -10.787363096221368) / Math.Sqrt((0.0169925438159128 / (27)) + (0.027553603728775383 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.990318414427223 = (0.98153326313645 - 10.787363096221368) / 0.98153326313645 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.738140463328905 > 1.389075512223066.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -195.0086372217546 (T) = (0 -10.736437126857687) / Math.Sqrt((0.03935414562649254 / (27)) + (0.010086899789770764 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.702008393404724 = (1.2337884131432075 - 10.736437126857687) / 1.2337884131432075 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.606815059822361 > 1.4789090616702483.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.89278518484567 (T) = (0 -9.746185343356151) / Math.Sqrt((0.12850685574305099 / (27)) + (0.404301630183475 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.051268362483137 = (1.2105155243328984 - 9.746185343356151) / 1.2105155243328984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.81354379070128 > 1.2797271746154895.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -84.11989565283358 (T) = (0 -10.809237449984979) / Math.Sqrt((0.22320156424054802 / (27)) + (0.05040791942767903 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.485411634523855 = (1.2738612946019467 - 10.809237449984979) / 1.2738612946019467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.670632156770381 > 1.4559217963615163.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.31184529893736 (T) = (0 -10.709674250499365) / Math.Sqrt((0.05769152238103386 / (27)) + (0.008873472658650105 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.9362154333532375 = (1.1984574824065821 - 10.709674250499365) / 1.1984574824065821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.94924400775206 > 2.0780566499115856.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -274.34896747777555 (T) = (0 -32.85241627341654) / Math.Sqrt((0.32257034551460884 / (27)) + (0.011312620779836375 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.51548485840197 = (1.6013472993772344 - 32.85241627341654) / 1.6013472993772344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.076079002743407 > 0.9648970421856569.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -155.99312084816356 (T) = (0 -9.056391648555104) / Math.Sqrt((0.02648174129103561 / (27)) + (0.018429342140928714 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.90705655440662 = (1.016765930836557 - 9.056391648555104) / 1.016765930836557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementBenchmark - Duration of single invocation 1.37 ns 9.08 ns 6.62 0.14 False
AndNotBenchmark - Duration of single invocation 1.79 ns 17.79 ns 9.96 0.07 False
AddBenchmark - Duration of single invocation 1.72 ns 38.10 ns 22.12 0.07 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.89 ns 38.11 ns 42.97 0.06 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.68 ns 9.59 ns 5.69 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.89 ns 8.92 ns 10.05 0.10 False
NegateBenchmark - Duration of single invocation 1.36 ns 38.19 ns 27.99 0.06 False
SubtractBenchmark - Duration of single invocation 1.54 ns 38.04 ns 24.72 0.07 False
ConditionalSelectBenchmark - Duration of single invocation 2.26 ns 32.65 ns 14.44 0.35 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.67 ns 9.60 ns 5.76 0.10 False
BitwiseOrBenchmark - Duration of single invocation 1.12 ns 9.68 ns 8.62 0.35 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.13 ns 9.76 ns 8.63 0.08 False
SubtractionOperatorBenchmark - Duration of single invocation 1.13 ns 38.06 ns 33.67 0.07 False
XorBenchmark - Duration of single invocation 1.30 ns 9.84 ns 7.57 0.35 False
OneBenchmark - Duration of single invocation 1.24 ns 18.92 ns 15.30 0.17 False
BitwiseAndBenchmark - Duration of single invocation 1.32 ns 9.60 ns 7.27 0.34 False
AddOperatorBenchmark - Duration of single invocation 1.53 ns 38.37 ns 25.06 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.OnesComplementBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.084448308483756 > 0.9536509549429405.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -192.03456297807134 (T) = (0 -9.019813714934841) / Math.Sqrt((0.02090190450922008 / (27)) + (0.01060128698329298 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.893026041882721 = (1.014256977597389 - 9.019813714934841) / 1.014256977597389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.793371131859498 > 1.5149732068337414.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -173.78133072815697 (T) = (0 -17.795640698281964) / Math.Sqrt((0.06181695653264334 / (27)) + (0.07475536533188296 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.45380229619966 = (1.2312082546584282 - 17.795640698281964) / 1.2312082546584282 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.100799549032615 > 1.5243940512087688.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -283.25953804783785 (T) = (0 -38.23119153040255) / Math.Sqrt((0.057478536727738234 / (27)) + (0.16387363562637167 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.104266865477268 = (1.2699592287445807 - 38.23119153040255) / 1.2699592287445807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.10794785213533 > 0.9847351086854895.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -456.73885858094565 (T) = (0 -38.14962132578353) / Math.Sqrt((0.07599905602191183 / (27)) + (0.04151404687137657 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -34.47077337884378 = (1.0755226822468305 - 38.14962132578353) / 1.0755226822468305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.59028705473065 > 1.564974391695773.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -162.09797479960875 (T) = (0 -9.53230200657607) / Math.Sqrt((0.06412379007466779 / (27)) + (0.0027476991391828624 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.764648944694496 = (1.227653957631838 - 9.53230200657607) / 1.227653957631838 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.921693810987165 > 0.9355735212659904.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -234.36247303492752 (T) = (0 -9.02255617809921) / Math.Sqrt((0.014528787898642499 / (27)) + (0.00700209540813705 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.111974945823722 = (0.9901866754182094 - 9.02255617809921) / 0.9901866754182094 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.192342126453575 > 1.2020538086781198.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -402.9767328420838 (T) = (0 -38.253803755316504) / Math.Sqrt((0.026221810616009585 / (27)) + (0.08305351747931797 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -35.28595302669755 = (1.054231749877731 - 38.253803755316504) / 1.054231749877731 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.04271249409137 > 1.6002928544201354.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -277.75394671503693 (T) = (0 -38.46967030025007) / Math.Sqrt((0.06019234708390745 / (27)) + (0.17296020814791782 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.68370407761001 = (1.2537492280249667 - 38.46967030025007) / 1.2537492280249667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.64983193494897 > 2.3568209843026304.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -189.71099990356083 (T) = (0 -32.96566107548993) / Math.Sqrt((0.27362448899105674 / (27)) + (0.18817045313918027 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -18.926615320812875 = (1.654353262947676 - 32.96566107548993) / 1.654353262947676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.598269420264508 > 1.6816237801875213.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -144.12097725081205 (T) = (0 -9.548970292813136) / Math.Sqrt((0.08132763225010306 / (27)) + (0.0028112488102922796 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.286707510002674 = (1.3104643324443843 - 9.548970292813136) / 1.3104643324443843 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.677033546102786 > 1.3764236195073534.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -144.14270571039023 (T) = (0 -9.608432175163502) / Math.Sqrt((0.05205401210619657 / (27)) + (0.01576813383572264 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.6781753781207644 = (1.2513952471759207 - 9.608432175163502) / 1.2513952471759207 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.762065890588294 > 1.6789806531421962.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -127.28831044261601 (T) = (0 -9.548612125389518) / Math.Sqrt((0.06380179369704184 / (27)) + (0.020798891729277267 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.659209522630718 = (1.2466837598809863 - 9.548612125389518) / 1.2466837598809863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.062240993750336 > 1.4040600895492132.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -231.20072897965076 (T) = (0 -38.37389126128821) / Math.Sqrt((0.0661405938833039 / (27)) + (0.2566319911087644 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.648297706775203 = (1.2520725173197846 - 38.37389126128821) / 1.2520725173197846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.842621228370701 > 1.490736904459282.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -133.0586114511129 (T) = (0 -9.654787442422217) / Math.Sqrt((0.040944305111844555 / (27)) + (0.026986125318749127 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.594339509617695 = (1.2713136448792037 - 9.654787442422217) / 1.2713136448792037 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.92200397844989 > 1.0706020552379636.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -63.82247194406942 (T) = (0 -20.08554239126748) / Math.Sqrt((0.04699502388072251 / (27)) + (0.9668797779162664 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.552720200043684 = (0.9772692955370837 - 20.08554239126748) / 0.9772692955370837 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.5978965639225 > 1.3738644487920246.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -117.22116165153062 (T) = (0 -9.593729914749073) / Math.Sqrt((0.05453659943505765 / (27)) + (0.03322198910853082 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.543381075377341 = (1.2718076707094061 - 9.593729914749073) / 1.2718076707094061 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.373573125777746 > 1.678898609926106.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -214.51682359421008 (T) = (0 -38.452445357263045) / Math.Sqrt((0.06139152282781802 / (27)) + (0.3061670943418278 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -30.242888890682668 = (1.2307583172544083 - 38.452445357263045) / 1.2307583172544083 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementBenchmark - Duration of single invocation 1.37 ns 11.07 ns 8.06 0.18 False
SubtractBenchmark - Duration of single invocation 1.16 ns 15.77 ns 13.55 0.10 False
BitwiseOrBenchmark - Duration of single invocation 1.46 ns 13.15 ns 9.03 0.10 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.50 ns 13.18 ns 8.77 0.15 False
AddBenchmark - Duration of single invocation 1.46 ns 15.72 ns 10.74 0.17 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.92 ns 14.48 ns 15.76 0.15 False
AddOperatorBenchmark - Duration of single invocation 1.13 ns 16.01 ns 14.16 0.15 False
OnesComplementOperatorBenchmark - Duration of single invocation 1.38 ns 11.10 ns 8.04 0.14 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.06 ns 13.28 ns 12.56 0.17 False
BitwiseAndBenchmark - Duration of single invocation 1.49 ns 13.32 ns 8.94 0.14 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.45 ns 13.49 ns 9.27 0.14 False
XorBenchmark - Duration of single invocation 1.04 ns 13.25 ns 12.75 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 1.42 ns 15.57 ns 10.93 0.11 False
NegateBenchmark - Duration of single invocation 0.89 ns 14.48 ns 16.19 0.15 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.073369193067862 > 1.0090385398591017.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -190.77741068464448 (T) = (0 -11.073481662410787) / Math.Sqrt((0.043943883767073956 / (27)) + (0.012218008138085304 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.155686596057066 = (1.0903725275167564 - 11.073481662410787) / 1.0903725275167564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.771255473176078 > 1.2102398639259817.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -212.8568978274401 (T) = (0 -15.773224065145984) / Math.Sqrt((0.1185105999820854 / (27)) + (0.0030073205964862067 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.735223445633933 = (1.2385510260170252 - 15.773224065145984) / 1.2385510260170252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.147399607691666 > 1.334460836959069.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -76.91622352643809 (T) = (0 -13.422980643373124) / Math.Sqrt((0.0740297064416229 / (27)) + (0.2445147460483139 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.580552929910182 = (1.2686464244630995 - 13.422980643373124) / 1.2686464244630995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.1761965373966 > 1.58503861104502.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -262.34861185394107 (T) = (0 -13.240210135501966) / Math.Sqrt((0.04206660131250429 / (27)) + (0.005834120754901919 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.58246738714105 = (1.2511458482348257 - 13.240210135501966) / 1.2511458482348257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.716183023100964 > 1.353937510929418.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -289.3930944875168 (T) = (0 -15.713096294918374) / Math.Sqrt((0.05824009831227207 / (27)) + (0.003907699796325303 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.008284043942533 = (1.2079299807598658 - 15.713096294918374) / 1.2079299807598658 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.483031915749923 > 0.9791016701955975.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -60.75181022485823 (T) = (0 -14.802561173065941) / Math.Sqrt((0.04319827199947953 / (27)) + (0.5415766505774781 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.393640566312248 = (1.105193251959995 - 14.802561173065941) / 1.105193251959995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.014116186979763 > 1.5711344029973788.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -245.61299485341922 (T) = (0 -15.853190313509545) / Math.Sqrt((0.03826723593741795 / (27)) + (0.023459801364521103 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.004247883099525 = (1.2190778317992954 - 15.853190313509545) / 1.2190778317992954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.099041913022894 > 0.9943283688429907.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -167.30675568621768 (T) = (0 -11.093951397738909) / Math.Sqrt((0.07650136054008018 / (27)) + (0.008542403974675757 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.650251602909973 = (1.0416609683387859 - 11.093951397738909) / 1.0416609683387859 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.280517597280936 > 1.1942915178215956.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -93.66232225470023 (T) = (0 -13.480878830484125) / Math.Sqrt((0.03572021710957241 / (27)) + (0.1738211790387246 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.013537150671652 = (1.2240280888925865 - 13.480878830484125) / 1.2240280888925865 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.318005614583246 > 1.1850541631375968.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -75.70136126538372 (T) = (0 -13.504025954980248) / Math.Sqrt((0.14873512861940993 / (27)) + (0.23257014596051614 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.788008058245817 = (1.1455731866024694 - 13.504025954980248) / 1.1455731866024694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.486643801800989 > 1.5534469198496448.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -159.06140630538692 (T) = (0 -13.346015758701979) / Math.Sqrt((0.05404942283621373 / (27)) + (0.04126150779887682 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.413795618032028 = (1.2815707402200847 - 13.346015758701979) / 1.2815707402200847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.254118752595453 > 1.3840617226213652.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -194.7823530795493 (T) = (0 -13.314672735720189) / Math.Sqrt((0.04534444727769682 / (27)) + (0.02363623414536777 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.541643965034359 = (1.2630546791263018 - 13.314672735720189) / 1.2630546791263018 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.571030226451857 > 1.3378406582998978.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -104.0361385710311 (T) = (0 -15.875518971604428) / Math.Sqrt((0.03711207037973473 / (27)) + (0.20329799717994312 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.059914360303276 = (1.2155913533291935 - 15.875518971604428) / 1.2155913533291935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.476850453221486 > 1.214030415251304.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -193.08445322731436 (T) = (0 -14.553131720748853) / Math.Sqrt((0.03753272643819231 / (27)) + (0.03813319352707136 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.266365032501781 = (1.0969946692326475 - 14.553131720748853) / 1.0969946692326475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 1.30 ns 13.48 ns 10.35 0.09 False
BitwiseOrBenchmark - Duration of single invocation 1.14 ns 13.68 ns 11.95 0.07 False
SubtractBenchmark - Duration of single invocation 1.13 ns 34.92 ns 30.89 0.06 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.89 ns 32.94 ns 36.85 0.03 False
OnesComplementBenchmark - Duration of single invocation 0.92 ns 11.34 ns 12.31 0.13 False
AddOperatorBenchmark - Duration of single invocation 1.13 ns 36.44 ns 32.25 0.07 False
AddBenchmark - Duration of single invocation 1.11 ns 36.46 ns 32.97 0.07 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.27 ns 13.65 ns 10.72 0.11 False
SubtractionOperatorBenchmark - Duration of single invocation 1.27 ns 36.73 ns 28.85 0.07 False
XorBenchmark - Duration of single invocation 1.22 ns 13.55 ns 11.09 0.07 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.18 ns 13.52 ns 11.49 0.08 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.96 ns 11.10 ns 11.57 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.28 ns 13.49 ns 10.57 0.08 False
NegateBenchmark - Duration of single invocation 0.95 ns 32.85 ns 34.49 0.04 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.48116090397676 > 1.3342829938737715.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -60.03173849826535 (T) = (0 -13.62662040734236) / Math.Sqrt((0.09110177391776177 / (27)) + (0.43411192669887966 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.341142937882193 = (1.2015209121318904 - 13.62662040734236) / 1.2015209121318904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.682611777042156 > 1.3362485716047217.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -328.05840250601454 (T) = (0 -13.48744729923614) / Math.Sqrt((0.012412721480537937 / (27)) + (0.01047950362889782 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.643863989503654 = (1.1583308866708149 - 13.48744729923614) / 1.1583308866708149 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.91731040311678 > 1.2297851584428408.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -134.12039055434968 (T) = (0 -35.54494151656669) / Math.Sqrt((0.011610724903764587 / (27)) + (0.7185605574261837 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.824538784038538 = (1.153137821967103 - 35.54494151656669) / 1.153137821967103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.9363508706033 > 0.972569236536994.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -490.1377678672661 (T) = (0 -32.870367704877) / Math.Sqrt((0.009416313381422042 / (27)) + (0.042700376584944646 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -32.19272860144011 = (0.9902882073831939 - 32.870367704877) / 0.9902882073831939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.344990430683954 > 0.991191167092631.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -234.8131692046481 (T) = (0 -11.127311050736676) / Math.Sqrt((0.01702798260000549 / (27)) + (0.013469446385030287 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.978698398631847 = (1.0135364545694552 - 11.127311050736676) / 1.0135364545694552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.43680849778367 > 1.3101558374824864.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -172.49410244357912 (T) = (0 -36.2020878389973) / Math.Sqrt((0.008926989729903821 / (27)) + (0.4488473347843465 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -28.74044833683028 = (1.2172677233706997 - 36.2020878389973) / 1.2172677233706997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.455584516712584 > 1.2991788725148419.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -174.96919848336765 (T) = (0 -36.17143523570889) / Math.Sqrt((0.052632094305884586 / (27)) + (0.4192948342338362 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -30.50056234642055 = (1.1482790319081113 - 36.17143523570889) / 1.1482790319081113 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.646362120819365 > 1.268869880625395.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -166.1646777769222 (T) = (0 -13.473485272633111) / Math.Sqrt((0.05470556746946857 / (27)) + (0.038379863513541876 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.888147085035476 = (1.1333545233128233 - 13.473485272633111) / 1.1333545233128233 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.72682939327025 > 1.3207557388404367.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -117.06652445035074 (T) = (0 -36.073345389747345) / Math.Sqrt((0.05667475969493519 / (27)) + (0.9561410676018218 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -30.507093563812447 = (1.1449277387864008 - 36.073345389747345) / 1.1449277387864008 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.54624701799582 > 1.3045700057946352.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -168.35789628447566 (T) = (0 -13.483857695432862) / Math.Sqrt((0.06393820681639699 / (27)) + (0.0333006479200866 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.067336516060063 = (1.1173847416526084 - 13.483857695432862) / 1.1173847416526084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.524875917373212 > 1.2121290353418943.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -163.3653257814542 (T) = (0 -13.400946840210587) / Math.Sqrt((0.11286212880092682 / (27)) + (0.015789968876823886 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.56385563440879 = (1.158864937775204 - 13.400946840210587) / 1.158864937775204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.097442838415882 > 0.9715708958598782.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.445783869825355 (T) = (0 -11.356512999415154) / Math.Sqrt((0.016469103870924977 / (27)) + (0.42137535009885047 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.24015588700566 = (1.0103519126940232 - 11.356512999415154) / 1.0103519126940232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.490544358391215 > 1.315618380474348.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -258.6267893561612 (T) = (0 -13.395906038101145) / Math.Sqrt((0.012583750428232732 / (27)) + (0.019544458810619468 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.671835254141985 = (1.1477120561093714 - 13.395906038101145) / 1.1477120561093714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.84780377658209 > 0.9857953748334601.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -417.1377716881239 (T) = (0 -32.94129646227589) / Math.Sqrt((0.05176709555264206 / (27)) + (0.04350215604975784 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -32.739447949265916 = (0.9763436708214608 - 32.94129646227589) / 0.9763436708214608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementOperatorBenchmark - Duration of single invocation 0.14 ns 11.23 ns 82.17 0.14 False
SubtractionOperatorBenchmark - Duration of single invocation 0.09 ns 15.91 ns 175.57 0.08 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.96 ns 14.96 ns 15.63 0.06 False
BitwiseAndBenchmark - Duration of single invocation 1.28 ns 13.44 ns 10.51 0.08 False
AddBenchmark - Duration of single invocation 1.30 ns 15.94 ns 12.30 0.10 False
BitwiseOrBenchmark - Duration of single invocation 1.13 ns 13.26 ns 11.69 0.09 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.10 ns 13.63 ns 12.36 0.10 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.09 ns 13.49 ns 12.40 0.09 False
XorBenchmark - Duration of single invocation 1.15 ns 13.34 ns 11.60 0.06 False
OnesComplementBenchmark - Duration of single invocation 0.92 ns 11.18 ns 12.10 0.16 False
AddOperatorBenchmark - Duration of single invocation 1.28 ns 16.04 ns 12.53 0.08 False
SubtractBenchmark - Duration of single invocation 1.24 ns 15.96 ns 12.90 0.06 False
NegateBenchmark - Duration of single invocation 0.99 ns 14.74 ns 14.96 0.08 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.27 ns 13.52 ns 10.60 0.08 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.233298208094482 > 0.9365294688482788.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -245.22712153118306 (T) = (0 -11.121796712376453) / Math.Sqrt((0.03211807503041736 / (27)) + (0.005904919828385317 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.924199994399466 = (0.9327079986582006 - 11.121796712376453) / 0.9327079986582006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.907933323874696 > 1.1647655452916204.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -183.84290520431153 (T) = (0 -15.879012955976334) / Math.Sqrt((0.1415988812476073 / (27)) + (0.0140502249077609 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.38056136027474 = (1.0324078935759138 - 15.879012955976334) / 1.0324078935759138 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.95802489823152 > 0.9684175835592499.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -365.9244876266813 (T) = (0 -14.768538789057546) / Math.Sqrt((0.010231030231865262 / (27)) + (0.01142453322371613 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.895309183541045 = (0.9914892404768894 - 14.768538789057546) / 0.9914892404768894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.442457793281125 > 1.3220161118863927.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -204.84572507099932 (T) = (0 -13.474564054135051) / Math.Sqrt((0.01949904729056122 / (27)) + (0.031746272051551706 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.518583837982986 = (1.1698108242874565 - 13.474564054135051) / 1.1698108242874565 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.943998897392657 > 1.2970288916150168.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -75.89503873524596 (T) = (0 -16.148405852493212) / Math.Sqrt((0.06701987164471988 / (27)) + (0.40321474552401737 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.242441765479875 = (1.1338228457168713 - 16.148405852493212) / 1.1338228457168713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.256176780030476 > 1.2288603737217685.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -36.81286771144388 (T) = (0 -13.85507051019295) / Math.Sqrt((0.019230342162657964 / (27)) + (1.3039159535742155 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.125596474723952 = (1.1426300173417547 - 13.85507051019295) / 1.1426300173417547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.634943428742828 > 1.2269921476082493.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -145.61247629326672 (T) = (0 -13.494805173866759) / Math.Sqrt((0.10149995673003806 / (27)) + (0.03798711977189873 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.959845085220374 = (1.128342806926759 - 13.494805173866759) / 1.128342806926759 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.486340320941967 > 1.3139368360099313.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -152.2204809433002 (T) = (0 -13.432835061973146) / Math.Sqrt((0.13286826804443744 / (27)) + (0.018692239887592037 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.825638421819432 = (1.0473424105829094 - 13.432835061973146) / 1.0473424105829094 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.344858466482687 > 1.2139380488296574.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -211.39455261749941 (T) = (0 -13.389299057984088) / Math.Sqrt((0.039328786827537016 / (27)) + (0.02064561769487385 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.307279044929425 = (1.1841309482840006 - 13.389299057984088) / 1.1841309482840006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.184556459273608 > 0.9524526361236934.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -51.60410632441971 (T) = (0 -11.35023516234562) / Math.Sqrt((0.027437215040093485 / (27)) + (0.4297447888021429 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.143025603893928 = (1.0185954484731043 - 11.35023516234562) / 1.0185954484731043 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.041582527004515 > 1.272959326410534.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -155.66963717931185 (T) = (0 -15.950957525761673) / Math.Sqrt((0.165945936552507 / (27)) + (0.030613622859758466 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.852898950456025 = (1.2410396741814989 - 15.950957525761673) / 1.2410396741814989 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.956542343229705 > 1.3162654025112006.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -87.95395842598734 (T) = (0 -16.166264356338445) / Math.Sqrt((0.014182568340187555 / (27)) + (0.3119757617093273 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.277930059777532 = (1.2175289584715063 - 16.166264356338445) / 1.2175289584715063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.744213676745405 > 1.007740592956115.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -48.20574105396277 (T) = (0 -15.15303509633011) / Math.Sqrt((0.02235168235467799 / (27)) + (0.9349035562023061 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.69434963060527 = (1.0312150913279954 - 15.15303509633011) / 1.0312150913279954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.51607411835027 > 1.2951522499015782.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -219.82357769608794 (T) = (0 -13.430331098404332) / Math.Sqrt((0.04512573529332009 / (27)) + (0.015562055854064621 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.020843726440244 = (1.2186300279518034 - 13.430331098404332) / 1.2186300279518034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrBenchmark - Duration of single invocation 1.14 ns 9.61 ns 8.40 0.35 False
ConditionalSelectBenchmark - Duration of single invocation 1.84 ns 32.68 ns 17.80 0.37 False
XorBenchmark - Duration of single invocation 1.31 ns 9.68 ns 7.39 0.32 False
AndNotBenchmark - Duration of single invocation 1.24 ns 17.79 ns 14.38 0.07 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.12 ns 9.61 ns 8.56 0.11 False
NegateBenchmark - Duration of single invocation 0.89 ns 7.42 ns 8.36 0.07 False
SubtractionOperatorBenchmark - Duration of single invocation 0.87 ns 7.32 ns 8.40 0.07 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.91 ns 8.96 ns 9.88 0.14 False
OneBenchmark - Duration of single invocation 0.80 ns 6.41 ns 8.00 0.39 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.33 ns 10.90 ns 8.19 0.12 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.90 ns 7.36 ns 8.14 0.06 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.10 ns 10.01 ns 9.13 0.10 False
AddOperatorBenchmark - Duration of single invocation 1.29 ns 7.37 ns 5.70 0.11 False
BitwiseAndBenchmark - Duration of single invocation 1.33 ns 9.63 ns 7.26 0.34 False
OnesComplementBenchmark - Duration of single invocation 0.92 ns 9.11 ns 9.96 0.08 False
SubtractBenchmark - Duration of single invocation 1.09 ns 7.34 ns 6.72 0.11 False
AddBenchmark - Duration of single invocation 1.12 ns 7.12 ns 6.34 0.09 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseOrBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.607751917466302 > 1.3098874407573589.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.350952250267824 (T) = (0 -9.805595191913145) / Math.Sqrt((0.06661642064600676 / (27)) + (0.4470022499588896 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.033871781267592 = (1.2205317011377557 - 9.805595191913145) / 1.2205317011377557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.67715042009868 > 1.9008843885903584.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -337.8539082426014 (T) = (0 -32.773353840902026) / Math.Sqrt((0.2043101872830835 / (27)) + (0.010582187447941678 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.85402145278076 = (1.571560378179811 - 32.773353840902026) / 1.571560378179811 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.675035361366453 > 1.2177582163682943.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -243.45428633754344 (T) = (0 -9.604400991809996) / Math.Sqrt((0.02738101557990438 / (27)) + (0.001999246265923112 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.101884323834295 = (1.1854527425867543 - 9.604400991809996) / 1.1854527425867543 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.79218023546107 > 1.3667640964775114.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -159.16430358263366 (T) = (0 -17.859959377369787) / Math.Sqrt((0.1898177900291435 / (27)) + (0.042671290114758065 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.455877085114382 = (1.2354808547563456 - 17.859959377369787) / 1.2354808547563456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.60631714161093 > 1.328998920001654.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -94.91126462273441 (T) = (0 -9.74099497252661) / Math.Sqrt((0.0306771791972833 / (27)) + (0.0766801365363526 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.149883234614036 = (1.1952312311855993 - 9.74099497252661) / 1.1952312311855993 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.417012531077428 > 0.9631700155715827.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -227.2508819269329 (T) = (0 -7.391388527586812) / Math.Sqrt((0.013298550710029248 / (27)) + (0.003247144794485597 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.294880988134351 = (1.0132294878572299 - 7.391388527586812) / 1.0132294878572299 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.317361791885809 > 1.2697662282044813.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -34.226084603107516 (T) = (0 -7.550302574546225) / Math.Sqrt((0.04407110138659099 / (27)) + (0.3620740001234681 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -5.351938471801106 = (1.188661163527505 - 7.550302574546225) / 1.188661163527505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.95747253443658 > 0.9579830204959827.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -50.123167468507376 (T) = (0 -9.203600989756625) / Math.Sqrt((0.02649890062969163 / (27)) + (0.2824661353302833 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.027408226898606 = (1.019517535756611 - 9.203600989756625) / 1.019517535756611 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.405216558165331 > 0.8371302775931733.
IsChangePoint: Marked as a change because one of 1/5/2023 9:23:01 PM, 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -64.64656624689015 (T) = (0 -6.24938734518775) / Math.Sqrt((0.04894938151937301 / (27)) + (0.05307185245157817 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -5.360600277053989 = (0.9825153402160103 - 6.24938734518775) / 0.9825153402160103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.8984951968392 > 1.361674691901933.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -36.289614526085 (T) = (0 -10.001912877948028) / Math.Sqrt((0.029871258814334757 / (27)) + (0.6407893629599697 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.619733936396159 = (1.160350534221912 - 10.001912877948028) / 1.160350534221912 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.355450573865024 > 0.9573083698622162.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -126.19211596308506 (T) = (0 -7.351598318028899) / Math.Sqrt((0.06546377279645818 / (27)) + (0.0014259301594083226 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.548265221464028 = (0.9739454169050801 - 7.351598318028899) / 0.9739454169050801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.01306503879874 > 1.1840124956978462.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -114.489200822447 (T) = (0 -9.61593390682676) / Math.Sqrt((0.09898246161745775 / (27)) + (0.019833452840545412 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.368196431302782 = (1.1491047068226774 - 9.61593390682676) / 1.1491047068226774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.367038161968173 > 1.3850906263076381.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -99.30100854605524 (T) = (0 -7.369245682331201) / Math.Sqrt((0.05125361265542794 / (27)) + (0.02058458954234731 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -4.7913915605600685 = (1.272448185426879 - 7.369245682331201) / 1.272448185426879 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.62822613155342 > 1.214085998851906.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -169.31408222503075 (T) = (0 -9.548112416594822) / Math.Sqrt((0.0170970578224438 / (27)) + (0.019912573966316828 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.100622256015165 = (1.1786887617805923 - 9.548112416594822) / 1.1786887617805923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.112616523416161 > 0.9642646271767158.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.54816382015284 (T) = (0 -9.028202150899181) / Math.Sqrt((0.0199607755139451 / (27)) + (0.01484761232252819 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.987440581917401 = (1.004535392318898 - 9.028202150899181) / 1.004535392318898 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.335506234374942 > 1.1759491991130604.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -104.58910168576472 (T) = (0 -7.311538058932124) / Math.Sqrt((0.07386899545837709 / (27)) + (0.007447847864865441 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -5.085946419378386 = (1.201380616111129 - 7.311538058932124) / 1.201380616111129 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.119286734282581 > 1.2795952539250235.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -40.23461200972538 (T) = (0 -7.514264044836986) / Math.Sqrt((0.03012868738416433 / (26)) + (0.25712079054236836 / (11))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (11) - 2, .025) and -5.198567375069855 = (1.2122581864736615 - 7.514264044836986) / 1.2122581864736615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 1.28 ns 15.95 ns 12.43 0.08 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.26 ns 13.63 ns 10.78 0.09 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.14 ns 13.44 ns 11.80 0.08 False
BitwiseOrBenchmark - Duration of single invocation 1.29 ns 13.43 ns 10.41 0.09 False
SubtractBenchmark - Duration of single invocation 1.30 ns 15.96 ns 12.31 0.09 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.93 ns 17.16 ns 18.49 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.28 ns 15.67 ns 12.21 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 1.22 ns 15.95 ns 13.13 0.08 False
OnesComplementBenchmark - Duration of single invocation 0.82 ns 11.16 ns 13.65 0.15 False
BitwiseAndBenchmark - Duration of single invocation 0.00 ns 13.39 ns ∞ 0.08 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.13 ns 13.24 ns 11.70 0.09 False
AbsBenchmark - Duration of single invocation 7.83 ns 9.85 ns 1.26 0.09 False
XorBenchmark - Duration of single invocation 1.12 ns 13.32 ns 11.86 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.93 ns 11.16 ns 11.97 0.14 False
NegateBenchmark - Duration of single invocation 0.94 ns 14.80 ns 15.82 0.06 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AddBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.949846246876314 > 1.2296846966265056.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -195.30208334408496 (T) = (0 -15.97731814061235) / Math.Sqrt((0.015328410090903001 / (27)) + (0.05719779485302341 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.950801853715156 = (1.1452616350046951 - 15.97731814061235) / 1.1452616350046951 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.625063820751132 > 1.3224745418533088.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -46.59459135277396 (T) = (0 -13.610263839896817) / Math.Sqrt((0.011321725678899691 / (27)) + (0.7788613830819597 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.582197823350372 = (1.1751020011467728 - 13.610263839896817) / 1.1751020011467728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.43935734687693 > 1.3213914130681779.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -233.77021189438645 (T) = (0 -13.404190689832637) / Math.Sqrt((0.014834626933563902 / (27)) + (0.023978932243489032 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.251470515148894 = (1.1913278954768924 - 13.404190689832637) / 1.1913278954768924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.425583249137192 > 1.2372256994608761.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -126.45387923572504 (T) = (0 -13.366706542409544) / Math.Sqrt((0.22329193773285783 / (27)) + (0.01060943429997273 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.002196031128518 = (1.2149125960481995 - 13.366706542409544) / 1.2149125960481995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.959277555155525 > 1.219102866526764.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -280.6719237691155 (T) = (0 -15.911990918189456) / Math.Sqrt((0.05537705988718432 / (27)) + (0.007979519407745609 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.170242687530433 = (1.1229159068808139 - 15.911990918189456) / 1.1229159068808139 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.160843687397204 > 0.9663440511338101.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -61.96810497678953 (T) = (0 -14.969488830309286) / Math.Sqrt((0.015089218296322832 / (27)) + (0.5536851963224088 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.125463861578968 = (0.9896879174948225 - 14.969488830309286) / 0.9896879174948225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.668019420543935 > 1.2133869603016798.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -343.22740781011674 (T) = (0 -15.967925562118635) / Math.Sqrt((0.007260656862013742 / (27)) + (0.017423241854690352 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.375574119044314 = (1.193812349286996 - 15.967925562118635) / 1.193812349286996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.952238368208103 > 1.2772024891681637.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -203.89576324071615 (T) = (0 -15.828291025454012) / Math.Sqrt((0.09422383040275285 / (27)) + (0.019170940330509197 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.66717881590941 = (1.0791639772118378 - 15.828291025454012) / 1.0791639772118378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.161739317840334 > 0.9676916791508039.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -188.34871111160663 (T) = (0 -11.139737307533041) / Math.Sqrt((0.06886047805438553 / (27)) + (0.003565648010535686 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.696121698604149 = (1.041474435447643 - 11.139737307533041) / 1.041474435447643 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.38753720523879 > 1.197812303720902.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -158.91680871729355 (T) = (0 -13.466892689438701) / Math.Sqrt((0.10617212851936285 / (27)) + (0.023267126056154823 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -11.147374505124702 = (1.1086257926564562 - 13.466892689438701) / 1.1086257926564562 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.23646998590951 > 1.2679025738712446.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -131.81399823902683 (T) = (0 -13.415195091048979) / Math.Sqrt((0.15549678456874816 / (27)) + (0.03034061453450936 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.731052905534703 = (1.2501285017549293 - 13.415195091048979) / 1.2501285017549293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AbsBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.847488151424061 > 8.107505561012346.
IsChangePoint: Marked as a change because one of 3/7/2023 7:00:18 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -21.968774122482884 (T) = (0 -9.871339533035545) / Math.Sqrt((0.1559351729597081 / (16)) + (0.014885195235219528 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.29400318274132475 = (7.628528016541098 - 9.871339533035545) / 7.628528016541098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.31602569331825 > 1.2608042502231755.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -286.87433876560783 (T) = (0 -13.318997361787872) / Math.Sqrt((0.010664957240042936 / (27)) + (0.015471753181740346 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.654630978687992 = (1.1428073000460839 - 13.318997361787872) / 1.1428073000460839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.157645487150173 > 0.9816671597515079.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -308.0150370046439 (T) = (0 -11.16148019668469) / Math.Sqrt((0.014113933750486758 / (27)) + (0.006215222924810472 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.130544738825511 = (1.0027793300853705 - 11.16148019668469) / 1.0027793300853705 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.802427015454498 > 0.968334632287959.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -61.93259482166936 (T) = (0 -14.989236405944858) / Math.Sqrt((0.008557806791196398 / (27)) + (0.5592835463953659 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -14.282041899055542 = (0.9808398972437852 - 14.989236405944858) / 0.9808398972437852 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Narrow_ushort - Duration of single invocation 35.58 μs 44.58 μs 1.25 0.01 False
Widen_ushort - Duration of single invocation 34.44 μs 62.34 μs 1.81 0.12 False
Widen_float - Duration of single invocation 39.52 μs 64.09 μs 1.62 0.01 False
Convert_double_ulong - Duration of single invocation 32.41 μs 41.14 μs 1.27 0.50 False
Widen_int - Duration of single invocation 26.36 μs 53.44 μs 2.03 0.01 False
Convert_int_float - Duration of single invocation 17.42 μs 26.42 μs 1.52 0.01 False
Narrow_short - Duration of single invocation 35.42 μs 44.55 μs 1.26 0.01 False
Narrow_ulong - Duration of single invocation 13.37 μs 21.70 μs 1.62 0.01 False
Narrow_uint - Duration of single invocation 20.50 μs 29.00 μs 1.41 0.01 False
Widen_sbyte - Duration of single invocation 49.31 μs 73.54 μs 1.49 0.01 False
Convert_double_long - Duration of single invocation 10.33 μs 18.30 μs 1.77 0.01 False
Convert_ulong_double - Duration of single invocation 49.29 μs 56.09 μs 1.14 0.03 False
Convert_long_double - Duration of single invocation 49.99 μs 56.26 μs 1.13 0.01 False
Convert_float_int - Duration of single invocation 16.26 μs 25.46 μs 1.57 0.01 False
Widen_byte - Duration of single invocation 49.54 μs 73.79 μs 1.49 0.01 False
Narrow_long - Duration of single invocation 13.23 μs 21.77 μs 1.65 0.01 False
Narrow_int - Duration of single invocation 20.34 μs 28.92 μs 1.42 0.01 False
Widen_short - Duration of single invocation 34.50 μs 61.51 μs 1.78 0.01 False
Widen_uint - Duration of single invocation 25.67 μs 53.02 μs 2.07 0.01 False
Narrow_double - Duration of single invocation 14.37 μs 22.98 μs 1.60 0.01 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorConvert.Narrow_ushort

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.5815354471917 > 37.24748579354933.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -226.56116488795712 (T) = (0 -44427.91362925953) / Math.Sqrt((6427.957892146269 / (27)) + (14674.093443938003 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.25343663586260595 = (35444.88198135725 - 44427.91362925953) / 35444.88198135725 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.34074396527385 > 36.024412709744496.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -251.96694827419032 (T) = (0 -61950.81293282564) / Math.Sqrt((4787.259223495565 / (27)) + (130100.24192954511 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.8038360740754368 = (34343.92616001915 - 61950.81293282564) / 34343.92616001915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.09443229591837 > 41.42270514393377.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -702.5288494112312 (T) = (0 -64065.094114408574) / Math.Sqrt((20217.56099428939 / (27)) + (5248.125990679908 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.6232348685627302 = (39467.54431854589 - 64065.094114408574) / 39467.54431854589 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.13606739035088 > 35.52585089674289.
IsChangePoint: Marked as a change because one of 3/16/2023 10:20:48 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -3.4405407727396944 (T) = (0 -45013.87791647108) / Math.Sqrt((39481124.381752186 / (27)) + (33815461.62569271 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.19444776580273296 = (37685.93253319817 - 45013.87791647108) / 37685.93253319817 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.441777761299434 > 27.650603897206864.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -682.6674538432344 (T) = (0 -53259.04820321908) / Math.Sqrt((1699.977872886283 / (27)) + (16419.896167627514 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -1.0225054383496195 = (26333.203952558408 - 53259.04820321908) / 26333.203952558408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.41753344376212 > 18.289399289426854.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -896.4923274003592 (T) = (0 -26368.17970062222) / Math.Sqrt((1192.7227911706536 / (27)) + (613.5173503639447 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.5149309145322216 = (17405.532785476327 - 26368.17970062222) / 17405.532785476327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_short

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.5475130223172 > 37.17763729166667.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -307.3166723162534 (T) = (0 -44582.93283969623) / Math.Sqrt((5074.389790218246 / (27)) + (7647.869636732479 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.2576328179155386 = (35449.880286672334 - 44582.93283969623) / 35449.880286672334 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.701598120344507 > 13.99840323479995.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -176.8686877232588 (T) = (0 -21795.068945321622) / Math.Sqrt((17894.201195398255 / (27)) + (18513.49793839599 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.6475570938710696 = (13228.718462261197 - 21795.068945321622) / 13228.718462261197 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.99848337924854 > 21.539840600123355.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -305.3978345741281 (T) = (0 -29126.92989672008) / Math.Sqrt((2089.8543166937907 / (27)) + (7837.826982541175 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.417817132643653 = (20543.50256186437 - 29126.92989672008) / 20543.50256186437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.54166447574336 > 51.91978572246311.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -369.6156859356229 (T) = (0 -73531.78461630952) / Math.Sqrt((30776.288964168034 / (27)) + (33636.75982355586 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.4829630290924601 = (49584.36803465648 - 73531.78461630952) / 49584.36803465648 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.2978919353263 > 10.845965890214105.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -890.0618006458952 (T) = (0 -18316.182826730565) / Math.Sqrt((758.5704221451799 / (27)) + (579.468120816248 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.7753737910212493 = (10316.803660931897 - 18316.182826730565) / 10316.803660931897 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.0896101702509 > 51.77016405923459.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -141.0138529060713 (T) = (0 -56117.590125768045) / Math.Sqrt((57540.28098908722 / (27)) + (1578.3799840647 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.13616222591373045 = (49392.23364923689 - 56117.590125768045) / 49392.23364923689 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_long_double

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.26377678571429 > 52.486596980946736.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -71.86994767039414 (T) = (0 -56095.334474911746) / Math.Sqrt((92219.24001110437 / (27)) + (40156.60675391392 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.12069795659742877 = (50053.92768380144 - 56095.334474911746) / 50053.92768380144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.4617555249183 > 17.108158337195253.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -343.7066450755207 (T) = (0 -25526.2931442991) / Math.Sqrt((7170.109014218092 / (27)) + (5076.953610907294 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.5700459399022327 = (16258.309706459055 - 25526.2931442991) / 16258.309706459055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_byte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.79002890995261 > 52.23984758465444.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -253.18191156668394 (T) = (0 -73780.50102295261) / Math.Sqrt((57804.385177619166 / (27)) + (75671.56390306671 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.48348132970226165 = (49734.701438919044 - 73780.50102295261) / 49734.701438919044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.768839675414366 > 13.912918812525188.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -224.57201481020493 (T) = (0 -21765.3053628289) / Math.Sqrt((24859.90009230357 / (27)) + (5324.9726961427095 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.6305894125954783 = (13348.121357039932 - 21765.3053628289) / 13348.121357039932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.91664715316145 > 21.348486561494155.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -294.63867144816743 (T) = (0 -28840.993305858352) / Math.Sqrt((4347.09224508185 / (27)) + (7347.815422277474 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.4167104561725489 = (20357.718953932566 - 28840.993305858352) / 20357.718953932566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_short

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.51224458661417 > 35.884902620613346.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -505.0473836495372 (T) = (0 -61676.57466823705) / Math.Sqrt((10379.49889598087 / (27)) + (28153.29676007106 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.7995020863313286 = (34274.24460172647 - 61676.57466823705) / 34274.24460172647 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.02132677970179 > 27.07518586705348.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -359.47587529320873 (T) = (0 -52902.597017528096) / Math.Sqrt((11027.969592785417 / (27)) + (57920.8875684372 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -1.0485204549648053 = (25824.78338906164 - 52902.597017528096) / 25824.78338906164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_double

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.979074307607085 > 15.08544856688224.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -175.44385129379185 (T) = (0 -22890.199235350443) / Math.Sqrt((9699.533631164279 / (27)) + (22037.320782276416 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -0.5937575206228991 = (14362.410177932281 - 22890.199235350443) / 14362.410177932281 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractBenchmark - Duration of single invocation 1.15 ns 38.41 ns 33.45 0.07 False
XorBenchmark - Duration of single invocation 1.27 ns 13.34 ns 10.50 0.09 False
BitwiseAndBenchmark - Duration of single invocation 1.12 ns 13.20 ns 11.79 0.08 False
OnesComplementBenchmark - Duration of single invocation 0.95 ns 11.09 ns 11.71 0.17 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.13 ns 13.66 ns 12.09 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.33 ns 38.39 ns 28.76 0.06 False
AllBitsSetBenchmark - Duration of single invocation 5.34 ns 6.68 ns 1.25 0.14 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.96 ns 11.06 ns 11.58 0.15 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.23 ns 13.36 ns 10.87 0.08 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.88 ns 35.73 ns 40.68 0.06 False
AddBenchmark - Duration of single invocation 1.19 ns 38.41 ns 32.28 0.05 False
BitwiseOrBenchmark - Duration of single invocation 0.02 ns 13.25 ns 740.64 0.07 False
NegateBenchmark - Duration of single invocation 0.93 ns 35.68 ns 38.25 0.05 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.32 ns 13.18 ns 10.00 0.08 False
SubtractionOperatorBenchmark - Duration of single invocation 1.19 ns 38.40 ns 32.20 0.09 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.SubtractBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.41358880144549 > 1.1983943428390358.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -910.6310551281406 (T) = (0 -38.33274702913615) / Math.Sqrt((0.015603501357852002 / (27)) + (0.011993704651697887 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -32.660664772925465 = (1.1387994648272253 - 38.33274702913615) / 1.1387994648272253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.337932429924004 > 1.341327036408231.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -181.77523673967764 (T) = (0 -13.331694056063414) / Math.Sqrt((0.05521463054675514 / (27)) + (0.026436391867573628 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.035284359969674 = (1.2080970114756562 - 13.331694056063414) / 1.2080970114756562 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.198055757863852 > 1.2244156606174337.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.49964924330044 (T) = (0 -13.622108207824924) / Math.Sqrt((0.010916683720799658 / (26)) + (0.6157672583647328 / (11))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (11) - 2, .025) and -10.801446556854637 = (1.1542744478144327 - 13.622108207824924) / 1.1542744478144327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.088242135520224 > 0.980357729662444.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -207.54322837126912 (T) = (0 -11.01762351255117) / Math.Sqrt((0.051401641584015396 / (27)) + (0.004539393198761137 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.710261324183007 = (1.0286979167981793 - 11.01762351255117) / 1.0286979167981793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.658919328806025 > 1.2036415104746254.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -248.5429048539873 (T) = (0 -13.284194189952563) / Math.Sqrt((0.014016957867140493 / (27)) + (0.02054981233219138 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.649143824152373 = (1.14035798600153 - 13.284194189952563) / 1.14035798600153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.38657423254857 > 1.3399669109661436.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -270.07980749942413 (T) = (0 -38.485838546843155) / Math.Sqrt((0.015117926714906598 / (27)) + (0.2039142444344248 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -32.10810630940719 = (1.1624294723225521 - 38.485838546843155) / 1.1624294723225521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.679975981454337 > 6.277683389413582.
IsChangePoint: Marked as a change because one of 12/15/2022 1:21:30 AM, 12/30/2022 6:06:02 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -10.077595456573178 (T) = (0 -6.4721592869558515) / Math.Sqrt((0.03581842844863469 / (16)) + (0.010212659306503174 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08808072976504787 = (5.948234455317853 - 6.4721592869558515) / 5.948234455317853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.05537281951836 > 0.9720146039488834.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.97334490276567 (T) = (0 -11.259423065571585) / Math.Sqrt((0.012980256859785459 / (27)) + (0.4075186186614874 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.289182105952664 = (0.9973639330022512 - 11.259423065571585) / 0.9973639330022512 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.356766825609453 > 1.2732332058978622.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -383.537729673798 (T) = (0 -13.275215720911179) / Math.Sqrt((0.007249213947002715 / (27)) + (0.007955965664839592 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.092567802612047 = (1.1967666961463304 - 13.275215720911179) / 1.1967666961463304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.72660129382237 > 0.9518737321283182.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -677.9682674150906 (T) = (0 -35.72154721030299) / Math.Sqrt((0.02394267202616326 / (27)) + (0.019071161956140247 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -34.167812062293805 = (1.0157455103271233 - 35.72154721030299) / 1.0157455103271233 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.40668432477739 > 1.2326265119708142.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -704.390252522778 (T) = (0 -38.33377730819459) / Math.Sqrt((0.05713599302527699 / (27)) + (0.00723648392041198 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -30.056159267341474 = (1.2343373492583234 - 38.33377730819459) / 1.2343373492583234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.251808918795867 > 1.2103477221168877.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -144.58597132262287 (T) = (0 -13.210837567446758) / Math.Sqrt((0.17360833746644394 / (27)) + (0.00555749990345797 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.290746901241032 = (1.1700587820274917 - 13.210837567446758) / 1.1700587820274917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.68089791064667 > 0.9706828112700638.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -857.14075618424 (T) = (0 -35.734207959159896) / Math.Sqrt((0.03478389645005991 / (27)) + (0.003924191497709376 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -35.86269406924546 = (0.9693867705934422 - 35.734207959159896) / 0.9693867705934422 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.177947282463466 > 1.308775688069438.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -54.547786835528896 (T) = (0 -13.47270896722967) / Math.Sqrt((0.015048607449563472 / (27)) + (0.5525505629924691 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.421749768282472 = (1.1795661120717764 - 13.47270896722967) / 1.1795661120717764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.39876983880281 > 1.2272191786470967.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -644.2507957345791 (T) = (0 -38.31447682976499) / Math.Sqrt((0.05611371283390133 / (27)) + (0.013861023348595162 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -34.13476359771326 = (1.0905004874505169 - 38.31447682976499) / 1.0905004874505169 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 1.51 ns 22.82 ns 15.10 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.12 ns 22.78 ns 20.35 0.07 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.86 ns 8.65 ns 10.02 0.10 False
BitwiseAndBenchmark - Duration of single invocation 1.30 ns 9.51 ns 7.31 0.34 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.17 ns 9.50 ns 8.13 0.10 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.48 ns 9.49 ns 6.39 0.11 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.95 ns 22.73 ns 23.88 0.05 False
XorBenchmark - Duration of single invocation 1.29 ns 9.49 ns 7.36 0.33 False
AndNotBenchmark - Duration of single invocation 1.31 ns 17.58 ns 13.45 0.09 False
AddBenchmark - Duration of single invocation 1.13 ns 22.86 ns 20.24 0.06 False
NegateBenchmark - Duration of single invocation 1.41 ns 23.59 ns 16.72 0.07 False
OnesComplementBenchmark - Duration of single invocation 0.94 ns 9.12 ns 9.76 0.11 False
SubtractBenchmark - Duration of single invocation 1.54 ns 22.87 ns 14.85 0.08 False
BitwiseOrBenchmark - Duration of single invocation 1.57 ns 9.61 ns 6.11 0.34 False
ConditionalSelectBenchmark - Duration of single invocation 1.89 ns 32.65 ns 17.28 0.39 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.32 ns 9.63 ns 7.29 0.10 False
OneBenchmark - Duration of single invocation 0.67 ns 12.30 ns 18.40 0.18 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.823799235954887 > 1.6313286115660224.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -268.92298640779626 (T) = (0 -22.90132162397708) / Math.Sqrt((0.06249244224066794 / (27)) + (0.04555071146611327 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -16.69279455825907 = (1.294386906973192 - 22.90132162397708) / 1.294386906973192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.782688045178197 > 1.363252353506729.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -452.97559007749953 (T) = (0 -22.8219367471403) / Math.Sqrt((0.04607226420844582 / (27)) + (0.006319608092064728 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -18.20346988526241 = (1.1884277624563495 - 22.8219367471403) / 1.1884277624563495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.645751469699233 > 0.9387611267949393.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -131.87293907035522 (T) = (0 -9.05854339173115) / Math.Sqrt((0.011427784172452651 / (27)) + (0.03672056366235178 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.332432132073153 = (0.9706519440521064 - 9.05854339173115) / 0.9706519440521064 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.506399039280009 > 1.5224644190825638.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -146.8773777097187 (T) = (0 -9.518347863621551) / Math.Sqrt((0.04710877337553457 / (27)) + (0.015478597681696382 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.480798344373003 = (1.2723705980901325 - 9.518347863621551) / 1.2723705980901325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.504757630781342 > 1.41387009110071.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -121.54821186244709 (T) = (0 -9.55341295259555) / Math.Sqrt((0.1133944986043648 / (27)) + (0.0052016889165607426 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.675002442188859 = (1.2447439625662167 - 9.55341295259555) / 1.2447439625662167 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.487895278475884 > 1.465413662759249.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -84.97342084573523 (T) = (0 -9.676850467384465) / Math.Sqrt((0.15711185135882733 / (27)) + (0.04385099971612956 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.664294318862535 = (1.2625885782555146 - 9.676850467384465) / 1.2625885782555146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.728614010258962 > 0.9711099369730236.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -433.02685577602665 (T) = (0 -22.759855581475552) / Math.Sqrt((0.014929075969423618 / (27)) + (0.021719283791090997 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -21.986325117977444 = (0.9901476405932861 - 22.759855581475552) / 0.9901476405932861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.490416001787876 > 1.6336575312868873.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -111.4851910690563 (T) = (0 -9.58046129271957) / Math.Sqrt((0.04681769800320834 / (27)) + (0.042178240643760544 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.595917203555757 = (1.2612645762166574 - 9.58046129271957) / 1.2612645762166574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.577813909554184 > 1.6222421406759855.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -71.01285593058694 (T) = (0 -17.83525414277086) / Math.Sqrt((0.07338179964591576 / (27)) + (0.564921656146692 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.491116138997503 = (1.3219998967480657 - 17.83525414277086) / 1.3219998967480657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.864445801436016 > 1.204970455760152.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -401.78441704877304 (T) = (0 -22.848144139701724) / Math.Sqrt((0.06939711103526522 / (27)) + (0.003865270071448884 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.20581522353674 = (1.1307707156050337 - 22.848144139701724) / 1.1307707156050337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.593902256932928 > 1.2024388650556774.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -238.87180919418805 (T) = (0 -22.777433502008382) / Math.Sqrt((0.029196562364178225 / (27)) + (0.07912551238575868 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.72334563149894 = (1.0485232748394433 - 22.777433502008382) / 1.0485232748394433 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.123891216802361 > 0.9512561407310466.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -227.8880392010636 (T) = (0 -9.061420889486758) / Math.Sqrt((0.018450173764690967 / (27)) + (0.006260185274906388 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.093448707135186 = (0.9964779239780286 - 9.061420889486758) / 0.9964779239780286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.867584077989687 > 1.6143682286717982.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -452.8903141653853 (T) = (0 -22.858658036713386) / Math.Sqrt((0.04982117584453042 / (27)) + (0.00476656814155483 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -17.429872887914645 = (1.2403047039843074 - 22.858658036713386) / 1.2403047039843074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.605356959460261 > 1.3617271247205982.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -101.46194261596277 (T) = (0 -9.634893710913788) / Math.Sqrt((0.0907173407271004 / (27)) + (0.03760030126420464 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.517835222174285 = (1.2816048006073766 - 9.634893710913788) / 1.2816048006073766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.64820015662553 > 2.1028760866910408.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -276.9079833110034 (T) = (0 -32.89375504637865) / Math.Sqrt((0.2396923035969622 / (27)) + (0.043217831741129045 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.121235813071056 = (1.5573783341797678 - 32.89375504637865) / 1.5573783341797678 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.625078146162029 > 1.3330755056465973.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -151.33758168122048 (T) = (0 -9.529996216310861) / Math.Sqrt((0.03961621714031409 / (27)) + (0.017036175271302408 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.819123931075074 = (1.2188061348454102 - 9.529996216310861) / 1.2188061348454102 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.296820149939808 > 1.0538736786408625.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -230.35528538730705 (T) = (0 -12.354307369259054) / Math.Sqrt((0.05294203830027119 / (27)) + (0.00485798329897409 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.616780654120198 = (1.063488046912315 - 12.354307369259054) / 1.063488046912315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 1.31 ns 9.45 ns 7.20 0.36 False
AddBenchmark - Duration of single invocation 1.29 ns 22.78 ns 17.67 0.07 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.89 ns 9.22 ns 10.32 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.08 ns 9.61 ns 8.88 0.13 False
SubtractionOperatorBenchmark - Duration of single invocation 1.26 ns 22.81 ns 18.17 0.07 False
BitwiseOrBenchmark - Duration of single invocation 1.32 ns 9.66 ns 7.34 0.32 False
OneBenchmark - Duration of single invocation 1.23 ns 12.37 ns 10.04 0.17 False
OnesComplementBenchmark - Duration of single invocation 0.94 ns 9.08 ns 9.71 0.10 False
XorBenchmark - Duration of single invocation 1.34 ns 9.63 ns 7.18 0.35 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.16 ns 9.54 ns 8.22 0.09 False
AndNotBenchmark - Duration of single invocation 1.29 ns 17.68 ns 13.65 0.07 False
SubtractBenchmark - Duration of single invocation 1.30 ns 22.84 ns 17.57 0.07 False
AddOperatorBenchmark - Duration of single invocation 1.25 ns 23.61 ns 18.83 0.08 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.91 ns 22.72 ns 24.94 0.07 False
ConditionalSelectBenchmark - Duration of single invocation 1.71 ns 32.82 ns 19.21 0.40 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.18 ns 9.53 ns 8.04 0.11 False
NegateBenchmark - Duration of single invocation 0.88 ns 22.64 ns 25.78 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.447804154178705 > 1.3765836359198447.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -70.31380907500862 (T) = (0 -9.68856048179485) / Math.Sqrt((0.1159866022950056 / (27)) + (0.10993881353703287 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.550768512178325 = (1.2831224353082162 - 9.68856048179485) / 1.2831224353082162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.784226475067292 > 1.3551501949237281.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -397.38664310255575 (T) = (0 -22.835526628163304) / Math.Sqrt((0.042203112137371115 / (27)) + (0.015379058923310396 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -17.855710114074654 = (1.2110669123576498 - 22.835526628163304) / 1.2110669123576498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.219354747494002 > 0.9576509838330467.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -213.74212863817706 (T) = (0 -9.103152851551386) / Math.Sqrt((0.013023495966030677 / (27)) + (0.01063578327752156 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.421356456644725 = (0.9662252875626083 - 9.103152851551386) / 0.9662252875626083 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.605874606048085 > 1.363775471693146.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -167.5113313746335 (T) = (0 -9.581476425746395) / Math.Sqrt((0.06368329134219165 / (27)) + (0.001307936855497668 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.704557962707234 = (1.2436114404128187 - 9.581476425746395) / 1.2436114404128187 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.80500619584969 > 1.2175096274614479.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -241.59688015564495 (T) = (0 -22.910427226993964) / Math.Sqrt((0.05547214769458085 / (27)) + (0.06629942693439558 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -18.23267289770393 = (1.1912242956999022 - 22.910427226993964) / 1.1912242956999022 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.662011190603256 > 1.2812449890880087.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -146.4040358543197 (T) = (0 -9.563167439967755) / Math.Sqrt((0.08362918372033015 / (27)) + (0.002361570457249172 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.406920220351358 = (1.1375351721332347 - 9.563167439967755) / 1.1375351721332347 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.37122798280552 > 0.8587481534920526.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -236.98333567297948 (T) = (0 -12.322015903788595) / Math.Sqrt((0.05186216186796643 / (27)) + (0.0037879614038569772 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.813439194653979 = (1.0430506900450096 - 12.322015903788595) / 1.0430506900450096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.084332864083873 > 0.9523590118847525.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -269.5358859845903 (T) = (0 -9.034794282164142) / Math.Sqrt((0.011699641569040013 / (27)) + (0.0050528898641440684 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.20324388128747 = (0.9816967146262605 - 9.034794282164142) / 0.9816967146262605 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.625532529235077 > 1.2901702416206802.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -151.47896228636796 (T) = (0 -9.586239377699806) / Math.Sqrt((0.0717305735809493 / (27)) + (0.004378234549131232 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.895859253792479 = (1.2140843788591364 - 9.586239377699806) / 1.2140843788591364 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.53752742943329 > 1.2088005974510918.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -182.22758018471242 (T) = (0 -9.572100422783258) / Math.Sqrt((0.04798392225869596 / (27)) + (0.0036091896877511313 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.905133789617505 = (1.2108714004758685 - 9.572100422783258) / 1.2108714004758685 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.67561569575626 > 1.3527431387135431.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -306.61028048612553 (T) = (0 -17.745554590101634) / Math.Sqrt((0.07240240825245853 / (27)) + (0.002121427403077568 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.576838227498426 = (1.307046183562821 - 17.745554590101634) / 1.307046183562821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.839768456036943 > 1.3831482514502997.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -283.7670142214343 (T) = (0 -22.909870616822573) / Math.Sqrt((0.056275720304160025 / (27)) + (0.04132309789401195 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -17.73716086460559 = (1.2226970127635086 - 22.909870616822573) / 1.2226970127635086 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.60861824653372 > 1.2147449852534777.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -259.7754882263673 (T) = (0 -22.876119951545835) / Math.Sqrt((0.03965850708817325 / (27)) + (0.06065271693406943 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -18.57581379875581 = (1.1685910065715779 - 22.876119951545835) / 1.1685910065715779 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.717642582566654 > 0.9547969180657639.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -608.5272931172293 (T) = (0 -22.674135225077762) / Math.Sqrt((0.022103360781427807 / (27)) + (0.00494612318252175 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -21.615067956739306 = (1.0026118545587148 - 22.674135225077762) / 1.0026118545587148 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.81836774877602 > 1.830530070298784.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -273.77015366969874 (T) = (0 -32.8210005847648) / Math.Sqrt((0.31599445563950057 / (27)) + (0.014085914805918004 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -19.190431507356482 = (1.6255720227081978 - 32.8210005847648) / 1.6255720227081978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.525948558762394 > 1.3108432844080797.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -188.98179635713043 (T) = (0 -9.585473057382238) / Math.Sqrt((0.04567369296251018 / (27)) + (0.00295177649857445 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.863556392550328 = (1.2189742883338635 - 9.585473057382238) / 1.2189742883338635 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.640968956420906 > 0.9621403063743137.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -246.47876923903797 (T) = (0 -22.760369408694206) / Math.Sqrt((0.016321570757410052 / (27)) + (0.07921311911139707 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -22.130265525427607 = (0.9840081335717151 - 22.760369408694206) / 0.9840081335717151 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrBenchmark - Duration of single invocation 1.12 ns 9.51 ns 8.49 0.36 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.69 ns 9.61 ns 5.70 0.12 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.31 ns 10.02 ns 7.66 0.12 False
BitwiseAndBenchmark - Duration of single invocation 1.32 ns 9.56 ns 7.27 0.36 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.91 ns 37.97 ns 41.66 0.03 False
SubtractBenchmark - Duration of single invocation 1.77 ns 38.19 ns 21.58 0.05 False
XorBenchmark - Duration of single invocation 1.29 ns 9.51 ns 7.36 0.37 False
NegateBenchmark - Duration of single invocation 0.90 ns 38.30 ns 42.34 0.06 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.91 ns 9.15 ns 10.01 0.11 False
AddBenchmark - Duration of single invocation 1.50 ns 38.74 ns 25.86 0.07 False
OnesComplementBenchmark - Duration of single invocation 0.90 ns 9.01 ns 10.07 0.11 False
AddOperatorBenchmark - Duration of single invocation 1.12 ns 37.69 ns 33.79 0.07 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.27 ns 9.61 ns 7.56 0.10 False
ConditionalSelectBenchmark - Duration of single invocation 1.71 ns 32.82 ns 19.23 0.39 False
SubtractionOperatorBenchmark - Duration of single invocation 1.60 ns 38.52 ns 24.00 0.07 False
OneBenchmark - Duration of single invocation 0.77 ns 20.62 ns 26.63 0.20 False
AndNotBenchmark - Duration of single invocation 1.30 ns 17.62 ns 13.60 0.08 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseOrBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.51064496700109 > 1.5859179038980111.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -178.86587963506742 (T) = (0 -9.552559810754586) / Math.Sqrt((0.049021194146198696 / (27)) + (0.0034455120030063117 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.349103062216853 = (1.299826622362411 - 9.552559810754586) / 1.299826622362411 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.61231127240301 > 1.19750993151903.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -98.10154264169181 (T) = (0 -9.649410829681312) / Math.Sqrt((0.1418689344351122 / (27)) + (0.02133268266939555 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.261537472263978 = (1.328838536816481 - 9.649410829681312) / 1.328838536816481 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.019825446728074 > 1.3505841687714435.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -90.77331704652703 (T) = (0 -9.53515983096421) / Math.Sqrt((0.08619851085096644 / (27)) + (0.05632228793692049 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.573778745842476 = (1.258969947623359 - 9.53515983096421) / 1.258969947623359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.558207822157112 > 1.6937921062599075.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -119.56731630913725 (T) = (0 -9.595561622577272) / Math.Sqrt((0.062418359468380404 / (27)) + (0.027662607386332173 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.445367811432608 = (1.2887961838289532 - 9.595561622577272) / 1.2887961838289532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.96976282100094 > 0.9767850262983429.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -459.25020732075933 (T) = (0 -38.19788740166421) / Math.Sqrt((0.016801034618248886 / (27)) + (0.06533299197222081 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -37.31846596514678 = (0.9968532518083516 - 38.19788740166421) / 0.9968532518083516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.186078612972864 > 1.2143752967587624.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -349.7891600823291 (T) = (0 -38.33215702568804) / Math.Sqrt((0.035149362099605876 / (27)) + (0.10973091678542032 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -31.31190649807467 = (1.1863167847422464 - 38.33215702568804) / 1.1863167847422464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.512630180306493 > 1.3597894419231114.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -148.9353938004412 (T) = (0 -9.545129613491286) / Math.Sqrt((0.019951315416751363 / (27)) + (0.02627090269282687 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.846671537434032 = (1.2164558651339537 - 9.545129613491286) / 1.2164558651339537 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.30072780606597 > 0.9541745503201892.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -423.6280970124585 (T) = (0 -38.264280147893324) / Math.Sqrt((0.06899941133616011 / (27)) + (0.056665242245472314 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -34.6156555964517 = (1.074366862187018 - 38.264280147893324) / 1.074366862187018 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.148564546408183 > 0.9454973744327914.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.271618169035634 (T) = (0 -9.26610886267366) / Math.Sqrt((0.033303046781278496 / (27)) + (0.4222297518151318 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.813974995149526 = (1.0512973848658467 - 9.26610886267366) / 1.0512973848658467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.74367636331698 > 1.30332844992061.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -230.6593327361667 (T) = (0 -38.45421882344704) / Math.Sqrt((0.09316821204571485 / (27)) + (0.24762694093683404 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -28.842910274601465 = (1.288554583638394 - 38.45421882344704) / 1.288554583638394 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.014379293976983 > 0.9580261743885566.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -40.290205284278784 (T) = (0 -9.211093382371082) / Math.Sqrt((0.16000198291742956 / (27)) + (0.38031723682226276 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.352334843172766 = (1.1028165842632935 - 9.211093382371082) / 1.1028165842632935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.692951098957856 > 1.4550046076006888.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -241.7171980590699 (T) = (0 -38.493492470584386) / Math.Sqrt((0.04860572026341491 / (27)) + (0.2420311348782259 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -31.056644240751073 = (1.2007960715254986 - 38.493492470584386) / 1.2007960715254986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.608426624497799 > 1.1961436388821953.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -149.6916541678355 (T) = (0 -9.49269080300869) / Math.Sqrt((0.03504135617745989 / (27)) + (0.01975706045818549 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.138567679266938 = (1.1663834690705333 - 9.49269080300869) / 1.1663834690705333 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.81720691506693 > 2.108172675266371.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -138.2360052316053 (T) = (0 -33.07913754383406) / Math.Sqrt((0.2084854583056414 / (27)) + (0.48642794193281347 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.01676936126085 = (1.5739401701199216 - 33.07913754383406) / 1.5739401701199216 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.519052924705925 > 1.4119534395038222.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -372.63225098632773 (T) = (0 -38.319523616868146) / Math.Sqrt((0.0402683578900529 / (27)) + (0.09262733992500449 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -30.397749608099982 = (1.2204544623472788 - 38.319523616868146) / 1.2204544623472788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.622440593992298 > 0.8255414580880523.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -91.50343915705614 (T) = (0 -20.491965107439913) / Math.Sqrt((0.035951417666644864 / (27)) + (0.48602030553210085 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.118009904431837 = (0.9703549340195857 - 20.491965107439913) / 0.9703549340195857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.622335500119505 > 1.5032711107709311.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -198.85484248821504 (T) = (0 -17.63896731562288) / Math.Sqrt((0.1596075596227464 / (27)) + (0.009274747988001342 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -12.611685661105604 = (1.2958694282828571 - 17.63896731562288) / 1.2958694282828571 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline dce343987e81ce6cf045d983ce62d8e117d2c142
Compare c6cc201665cb3c4d61851392bd8f8d8093688500
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.54 ns 13.15 ns 8.55 0.14 False
XorBenchmark - Duration of single invocation 1.02 ns 13.33 ns 13.03 0.15 False
SubtractionOperatorBenchmark - Duration of single invocation 1.13 ns 38.22 ns 33.76 0.16 False
OnesComplementBenchmark - Duration of single invocation 1.43 ns 11.04 ns 7.73 0.17 False
BitwiseAndBenchmark - Duration of single invocation 1.55 ns 13.27 ns 8.53 0.13 False
AddBenchmark - Duration of single invocation 1.48 ns 38.38 ns 25.86 0.15 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.85 ns 11.08 ns 13.09 0.16 False
NegateBenchmark - Duration of single invocation 0.88 ns 35.66 ns 40.48 0.14 False
BitwiseOrBenchmark - Duration of single invocation 1.10 ns 13.18 ns 11.97 0.10 False
AddOperatorBenchmark - Duration of single invocation 1.11 ns 38.23 ns 34.47 0.09 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.55 ns 13.31 ns 8.57 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.11 ns 13.29 ns 11.95 0.10 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.91 ns 35.73 ns 39.08 0.13 False
SubtractBenchmark - Duration of single invocation 1.47 ns 38.20 ns 26.02 0.15 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.153606609382383 > 1.5684526065195386.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -200.33286732188654 (T) = (0 -13.270205384391328) / Math.Sqrt((0.0693113302161327 / (27)) + (0.01152957746210299 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.834279299945948 = (1.2248350828889498 - 13.270205384391328) / 1.2248350828889498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.328606896077536 > 1.331742685682241.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -249.43085418145523 (T) = (0 -13.242290042856625) / Math.Sqrt((0.038863308302660854 / (27)) + (0.009814896114800535 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.053712981989468 = (1.1979947429821225 - 13.242290042856625) / 1.1979947429821225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.22080466592492 > 1.378651279699243.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -710.1805326496997 (T) = (0 -38.24634083050066) / Math.Sqrt((0.052510804588001204 / (27)) + (0.008476195955569473 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.865364776189704 = (1.2391345803891103 - 38.24634083050066) / 1.2391345803891103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.036295483489981 > 0.9940161215668392.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -169.264437710919 (T) = (0 -11.02827146079852) / Math.Sqrt((0.05649707053177791 / (27)) + (0.0150174978492333 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.257595457016654 = (1.075132228309383 - 11.02827146079852) / 1.075132228309383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.269629385958797 > 1.2008445493095679.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -181.06714963369737 (T) = (0 -13.331316387751105) / Math.Sqrt((0.07040618345896807 / (27)) + (0.02044854004647234 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.837319163473229 = (1.2301304581564598 - 13.331316387751105) / 1.2301304581564598 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.3796195224567 > 1.2070549814599278.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -465.4108796644186 (T) = (0 -38.3517313028541) / Math.Sqrt((0.033442255494255876 / (27)) + (0.05613380601125394 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -28.755279769774504 = (1.288905081706269 - 38.3517313028541) / 1.288905081706269 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.076263732835304 > 0.9758831674886375.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -43.29407323790544 (T) = (0 -11.315255274036533) / Math.Sqrt((0.04069434437688095 / (27)) + (0.5951409472272658 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.233929349415083 = (1.105660874499124 - 11.315255274036533) / 1.105660874499124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.66406944547829 > 0.9680053943865702.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -633.2776672056393 (T) = (0 -35.659797864755326) / Math.Sqrt((0.07018338170822695 / (27)) + (0.004298419335452311 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -33.59624603749583 = (1.0307418274834443 - 35.659797864755326) / 1.0307418274834443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.176134114818117 > 1.590048071260323.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -171.31528451661586 (T) = (0 -13.265583789801639) / Math.Sqrt((0.09875314547369562 / (27)) + (0.014360161033047913 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.08535394508865 = (1.1966766109149753 - 13.265583789801639) / 1.1966766109149753 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.22852147578387 > 1.57952683062291.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -457.8316942515632 (T) = (0 -38.29470744462524) / Math.Sqrt((0.14497255903543532 / (27)) + (0.012666532306365922 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -27.92429584719705 = (1.32396334372082 - 38.29470744462524) / 1.32396334372082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.313368427174716 > 1.586799092359671.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -176.11771932357883 (T) = (0 -13.365358556911326) / Math.Sqrt((0.09373615128263735 / (27)) + (0.014392581379973873 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.24224153009971 = (1.188851753551748 - 13.365358556911326) / 1.188851753551748 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.290158359072652 > 1.178491994041419.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -178.12010225211426 (T) = (0 -13.371237546143892) / Math.Sqrt((0.059811676568133944 / (27)) + (0.026924264608264283 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -10.066892821517134 = (1.2082196657897026 - 13.371237546143892) / 1.2082196657897026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.72666476011081 > 0.9409924296950982.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -1029.8540741822899 (T) = (0 -35.70157546481756) / Math.Sqrt((0.021561218914209646 / (27)) + (0.0036999578198503398 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -34.446094607547145 = (1.0072075883140033 - 35.70157546481756) / 1.0072075883140033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.203796375596994 > 1.1892207209671775.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -819.0002449061125 (T) = (0 -38.23256335097648) / Math.Sqrt((0.038777420482704494 / (27)) + (0.006646110900374709 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.88890346019316 = (1.2377442728016281 - 38.23256335097648) / 1.2377442728016281 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

@jandupej could be false positives as intrinsics were first introduced and then disabled in dotnet/runtime#83888.

@kotlarmilos kotlarmilos added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Mar 28, 2023
@kotlarmilos kotlarmilos mentioned this issue Mar 28, 2023
Closed
@jandupej
Copy link
Member

jandupej commented Mar 29, 2023 •
edited
Loading

@kotlarmilos yes, some intrinsics were unintentionally left enabled. We disabled them temporarily to prevent unexpected customer issues. When we're done with them, they will be reenabled and the performance improvements will return (and then some, hopefully).

@kotlarmilos
Copy link
Member

Thanks, closing this issue.

@matouskozak matouskozak mentioned this issue Oct 16, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kotlarmilos @jandupej