[Perf] Linux/x64: 100 Regressions on 5/11/2023 9:33:40 AM #17874

performanceautofiler · 2023-05-16T18:44:06Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
FixedTimeEquals_256Bit_SameReference - Duration of single invocation	50.07 ns	109.92 ns	2.20	0.02	True
FixedTimeEquals_256Bit_Equal - Duration of single invocation	49.97 ns	109.67 ns	2.19	0.04	True
FixedTimeEquals_256Bit_AllBitsDifferent - Duration of single invocation	51.02 ns	110.52 ns	2.17	0.04	True
FixedTimeEquals_256Bit_CascadingErrors - Duration of single invocation	50.19 ns	109.78 ns	2.19	0.05	True
FixedTimeEquals_256Bit_FirstBitDifferent - Duration of single invocation	49.89 ns	110.08 ns	2.21	0.04	True
FixedTimeEquals_256Bit_VersusZero - Duration of single invocation	50.27 ns	110.14 ns	2.19	0.04	True
FixedTimeEquals_256Bit_LastBitDifferent - Duration of single invocation	50.07 ns	109.76 ns	2.19	0.03	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_SameReference

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.92262977512158 > 52.58754456791563.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -53.468589604999345 (T) = (0 -109.98534547778553) / Math.Sqrt((25.723853387140995 / (23)) + (0.018291879086275907 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0593265254531927 = (53.408405184108055 - 109.98534547778553) / 53.408405184108055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_Equal

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.66657313911722 > 52.68785897710553.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -51.20783807695286 (T) = (0 -109.93719154136167) / Math.Sqrt((26.27237490422374 / (22)) + (0.19184354994111436 / (15))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (15) - 2, .025) and -1.0480594340886804 = (53.67871152151409 - 109.93719154136167) / 53.67871152151409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_AllBitsDifferent

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.51864327219187 > 53.37900490977008.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -52.43567006640001 (T) = (0 -109.96612698290058) / Math.Sqrt((25.857412668462516 / (23)) + (0.12127681798017742 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.030054801701211 = (54.16904356017759 - 109.96612698290058) / 54.16904356017759 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_CascadingErrors

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.77772828939396 > 52.78646802799586.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -52.006250405393054 (T) = (0 -109.96548105309935) / Math.Sqrt((26.769739749542634 / (23)) + (0.23658395774121338 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0561967913854893 = (53.48003727746474 - 109.96548105309935) / 53.48003727746474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_FirstBitDifferent

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.08100416942126 > 53.30648615198241.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -53.88517390254476 (T) = (0 -109.97879251348391) / Math.Sqrt((24.807855669825482 / (23)) + (0.08284713092918713 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0414522380942717 = (53.87282173995453 - 109.97879251348391) / 53.87282173995453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_VersusZero

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.14324307784193 > 52.36211017864677.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -48.913603166341694 (T) = (0 -110.01881413364569) / Math.Sqrt((30.4180154594816 / (23)) + (0.08498564572296184 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0507788464019123 = (53.647332244855896 - 110.01881413364569) / 53.647332244855896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_LastBitDifferent

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.75643311370456 > 52.68707326480414.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -55.42435125037183 (T) = (0 -109.93637005976296) / Math.Sqrt((23.873127455380242 / (23)) + (0.047215333353895605 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.05934083608089 = (53.384251957525265 - 109.93637005976296) / 53.384251957525265 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Xml.Linq.Perf_XElementList

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Enumerator - Duration of single invocation	182.21 ns	221.11 ns	1.21	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XElementList*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Xml.Linq.Perf_XElementList* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XElementList*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Xml.Linq.Perf_XElementList* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Xml.Linq.Perf_XElementList.Enumerator

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 221.1061538751201 > 191.64366627725903.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -26.592203439681263 (T) = (0 -219.09601895879968) / Math.Sqrt((35.42401924490699 / (23)) + (16.021592966446907 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.24380968748120463 = (176.14914979676942 - 219.09601895879968) / 176.14914979676942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

performanceautofiler · 2023-05-16T18:44:09Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in Exceptions.Handling

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
MultipleNestedTryCatch_FirstCatches - Duration of single invocation	5.61 μs	6.74 μs	1.20	0.02	True
ThrowAndCatchWhen - Duration of single invocation	5.30 μs	6.81 μs	1.29	0.01	True
MultipleNestedTryFinally - Duration of single invocation	5.67 μs	6.87 μs	1.21	0.01	True
ThrowAndCatchWhenFinally - Duration of single invocation	5.39 μs	6.70 μs	1.24	0.02	True
CatchAndThrowOtherDeep - Duration of single invocation	33.09 μs	198.66 μs	6.00	0.00	True
ThrowAndCatchDeep - Duration of single invocation	19.70 μs	39.83 μs	2.02	0.00	True
ThrowAndCatch_ManyCatchBlocks - Duration of single invocation	13.46 μs	29.63 μs	2.20	0.01	True
MultipleNestedTryFinally - Duration of single invocation	2.92 μs	9.22 μs	3.16	0.01	True
ThrowAndCatchWhenFinally - Duration of single invocation	2.66 μs	9.10 μs	3.43	0.01	True
ThrowAndCatchDeep - Duration of single invocation	12.69 μs	16.72 μs	1.32	0.01	True
CatchAndThrowOtherDeep - Duration of single invocation	30.25 μs	189.84 μs	6.28	0.00	True
CatchAndRethrowDeep - Duration of single invocation	37.70 μs	190.94 μs	5.06	0.00	True
ThrowAndCatch_ManyCatchBlocks - Duration of single invocation	10.37 μs	30.17 μs	2.91	0.00	True
ThrowAndCatchDeepRecursive - Duration of single invocation	16.78 μs	41.06 μs	2.45	0.00	True
ThrowAndCatch - Duration of single invocation	2.85 μs	8.92 μs	3.13	0.01	True
TryAndFinallyDeep - Duration of single invocation	10.32 μs	20.89 μs	2.02	0.00	True
CatchAndRethrowDeep - Duration of single invocation	41.17 μs	190.75 μs	4.63	0.01	True
ThrowAndCatchDeepRecursive - Duration of single invocation	19.95 μs	40.72 μs	2.04	0.00	True
ThrowAndCatchWhen - Duration of single invocation	2.63 μs	9.13 μs	3.47	0.01	True
ThrowAndCatch - Duration of single invocation	12.34 μs	29.31 μs	2.38	0.00	True
ThrowAndCatch - Duration of single invocation	9.35 μs	29.65 μs	3.17	0.00	True
TryAndFinallyDeep - Duration of single invocation	13.14 μs	17.82 μs	1.36	0.00	True
ThrowAndCatch - Duration of single invocation	5.59 μs	6.70 μs	1.20	0.01	True
TryAndCatchDeep_CaugtAtTheTop - Duration of single invocation	15.33 μs	18.68 μs	1.22	0.01	True
MultipleNestedTryCatch_FirstCatches - Duration of single invocation	2.86 μs	9.15 μs	3.20	0.01	True
ThrowAndCatchFinally - Duration of single invocation	5.62 μs	6.79 μs	1.21	0.02	True
MultipleNestedTryCatch_LastCatches - Duration of single invocation	6.29 μs	6.82 μs	1.08	0.01	False
ThrowAndCatchDeep - Duration of single invocation	9.82 μs	19.22 μs	1.96	0.00	True
ThrowAndCatchDeepRecursive - Duration of single invocation	10.16 μs	20.21 μs	1.99	0.00	True
ThrowAndCatchFinally - Duration of single invocation	2.84 μs	9.04 μs	3.18	0.01	True
ThrowAndCatchDeep - Duration of single invocation	16.45 μs	41.03 μs	2.49	0.00	True
MultipleNestedTryCatch_LastCatches - Duration of single invocation	3.59 μs	9.17 μs	2.56	0.00	True
ThrowAndCatchDeepRecursive - Duration of single invocation	13.08 μs	18.43 μs	1.41	0.00	True
ThrowAndCatch_ManyCatchBlocks - Duration of single invocation	3.85 μs	9.41 μs	2.44	0.02	True
TryAndCatchDeep_CaugtAtTheTop - Duration of single invocation	12.47 μs	20.36 μs	1.63	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Exceptions.Handling*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Exceptions.Handling* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Exceptions.Handling*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Exceptions.Handling* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

Exceptions.Handling.MultipleNestedTryCatch_FirstCatches(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.743548109515592 > 5.843245994258302.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -38.35536447159832 (T) = (0 -6769.607943806896) / Math.Sqrt((3445.57076165844 / (23)) + (11949.690809074658 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.21110235654221957 = (5589.62494560294 - 6769.607943806896) / 5589.62494560294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhen(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.810475871524762 > 5.587186791640989.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -47.141383977840356 (T) = (0 -6732.70764123078) / Math.Sqrt((5792.929396864952 / (23)) + (9129.92518012248 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.2584908551062766 = (5349.826432121526 - 6732.70764123078) / 5349.826432121526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.MultipleNestedTryFinally(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.870972645323427 > 5.963693184391176.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -43.956811037863375 (T) = (0 -6938.626305505971) / Math.Sqrt((6509.963644488112 / (23)) + (7227.6843424152785 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.21242542884637233 = (5722.930367856192 - 6938.626305505971) / 5722.930367856192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.697536751558773 > 5.6518557965776.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -44.13946206776754 (T) = (0 -6838.667378679485) / Math.Sqrt((4858.635118132985 / (23)) + (12702.713598035232 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.265743844238035 = (5402.884169502949 - 6838.667378679485) / 5402.884169502949 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndThrowOtherDeep(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 198.66421233974359 > 34.68033248052615.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -308.15614386832664 (T) = (0 -198060.21617104302) / Math.Sqrt((41573.44492442412 / (23)) + (4273501.652229174 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -4.9913424367519825 = (33057.73593512293 - 198060.21617104302) / 33057.73593512293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionHardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.833654746603 > 20.615432421664813.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -132.7131536882058 (T) = (0 -40084.49591056964) / Math.Sqrt((25283.509729117235 / (23)) + (338707.3915800401 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0386489385888122 = (19662.28473761491 - 40084.49591056964) / 19662.28473761491 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionHardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.627961668134997 > 14.203377324836286.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -226.9791636443847 (T) = (0 -29721.019099150293) / Math.Sqrt((21451.20994262048 / (23)) + (63237.914744066096 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.212284606197673 = (13434.5368655947 - 29721.019099150293) / 13434.5368655947 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.MultipleNestedTryFinally(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.220659618771752 > 3.071967071524081.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -216.74506268011626 (T) = (0 -9225.710700234447) / Math.Sqrt((675.1467891566576 / (23)) + (11335.160019947825 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -2.129958470858421 = (2947.5505142099246 - 9225.710700234447) / 2947.5505142099246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.099437920962913 > 2.782303157508247.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -150.71674889524388 (T) = (0 -9118.85212161401) / Math.Sqrt((470.7018235169969 / (23)) + (27394.961910117363 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.4516324497857838 = (2641.89546664505 - 9118.85212161401) / 2641.89546664505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.717045942764084 > 13.325183801704085.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -42.7969374029451 (T) = (0 -17063.07688950656) / Math.Sqrt((21793.153221583747 / (23)) + (138591.69736773433 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.33895698375989397 = (12743.558677734463 - 17063.07688950656) / 12743.558677734463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndThrowOtherDeep(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 189.83664487151566 > 31.223059498414813.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -266.46253222142684 (T) = (0 -193877.432385771) / Math.Sqrt((38208.25897573201 / (23)) + (5662281.853531561 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -5.505206203808604 = (29803.426103889153 - 193877.432385771) / 29803.426103889153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndRethrowDeep(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.94336028377109 > 39.44401826950176.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -709.930175693807 (T) = (0 -189507.86791969152) / Math.Sqrt((73918.68667333861 / (23)) + (638656.8853509629 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -4.041291516545896 = (37591.13459273532 - 189507.86791969152) / 37591.13459273532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionSoftware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.172328971228243 > 10.893410935248163.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -261.39635283366414 (T) = (0 -30184.47246640731) / Math.Sqrt((14201.218975506601 / (23)) + (76489.11590917816 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.8966268803889983 = (10420.55939988851 - 30184.47246640731) / 10420.55939988851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: ReflectionSoftware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.06463296775404 > 17.612136691017433.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -175.6543262719187 (T) = (0 -41592.68035424445) / Math.Sqrt((14024.704939809599 / (23)) + (270099.7945007074 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -1.473993240057249 = (16811.962005717516 - 41592.68035424445) / 16811.962005717516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.918743705897727 > 2.9549029832781066.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -179.26943511826624 (T) = (0 -9100.940257567216) / Math.Sqrt((699.3805162782774 / (23)) + (17913.15686464121 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.2187374385539504 = (2827.487619386533 - 9100.940257567216) / 2827.487619386533 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.TryAndFinallyDeep(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.8885758988737 > 10.726559223951039.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -104.16849440842599 (T) = (0 -20604.520366065306) / Math.Sqrt((13711.51335303398 / (22)) + (137967.7068994539 / (15))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (15) - 2, .025) and -1.0040829263082403 = (10281.271346401463 - 20604.520366065306) / 10281.271346401463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.CatchAndRethrowDeep(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.7498025035178 > 43.01228812740683.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -533.1489045398561 (T) = (0 -192024.60353493344) / Math.Sqrt((36854.56332900281 / (23)) + (1177459.6873221507 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -3.668337825508844 = (41133.39923380608 - 192024.60353493344) / 41133.39923380608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: ReflectionHardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.71814951277509 > 20.884988540703358.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -165.30113361696533 (T) = (0 -41023.80042005891) / Math.Sqrt((41455.611633453576 / (23)) + (219080.34572651918 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0666996042963564 = (19849.909650525227 - 41023.80042005891) / 19849.909650525227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhen(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.127751710958558 > 2.742827442244276.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -271.35502804904513 (T) = (0 -9125.829567901528) / Math.Sqrt((352.4631841584272 / (23)) + (8379.968143282664 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.476920666346765 = (2624.6873149079147 - 9125.829567901528) / 2624.6873149079147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch(kind: ReflectionHardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.311851098515245 > 13.043161607810944.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -219.63793563359073 (T) = (0 -29255.629334716552) / Math.Sqrt((14759.213475345889 / (23)) + (78758.88086459695 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.3600984243065937 = (12395.936132753435 - 29255.629334716552) / 12395.936132753435 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch(kind: ReflectionSoftware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.651712705912644 > 9.812586141832046.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -451.46632143027847 (T) = (0 -29827.600422398245) / Math.Sqrt((5724.076416992554 / (22)) + (26819.950915179325 / (15))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (15) - 2, .025) and -2.1746150147138827 = (9395.659090677647 - 29827.600422398245) / 9395.659090677647 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.TryAndFinallyDeep(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.822578666477916 > 13.777193233664756.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -49.63359290657714 (T) = (0 -18299.455376730268) / Math.Sqrt((9492.082069176859 / (23)) + (155149.10966525608 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3913919535057289 = (13151.90542149051 - 18299.455376730268) / 13151.90542149051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.703629431301506 > 5.8686015502674325.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -37.69260763751343 (T) = (0 -6718.635134929852) / Math.Sqrt((2531.3158590293683 / (23)) + (12287.745493413599 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.20629726782019134 = (5569.634711243763 - 6718.635134929852) / 5569.634711243763 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.TryAndCatchDeep_CaugtAtTheTop(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.6806159863001 > 16.083159712972005.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -27.40221358700588 (T) = (0 -18331.902892376285) / Math.Sqrt((9385.757807745256 / (23)) + (171463.3029431935 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.19423352649777406 = (15350.3502335399 - 18331.902892376285) / 15350.3502335399 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.MultipleNestedTryCatch_FirstCatches(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.154928288778528 > 2.9546732755258396.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -241.762099113859 (T) = (0 -9062.00397318157) / Math.Sqrt((801.6682244624819 / (23)) + (9428.838190754339 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.196657176126035 = (2834.837605001995 - 9062.00397318157) / 2834.837605001995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchFinally(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.791439032421478 > 5.866415754951881.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -37.23243075772719 (T) = (0 -6768.822291299832) / Math.Sqrt((2343.5001580225025 / (23)) + (13287.850933483172 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.20900662198298875 = (5598.664364797062 - 6768.822291299832) / 5598.664364797062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.MultipleNestedTryCatch_LastCatches(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.818010021350931 > 6.613640669319981.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -21.357787822219237 (T) = (0 -6934.268888039822) / Math.Sqrt((7021.532697094305 / (23)) + (7842.431676875435 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.09725401000207798 = (6319.656911553861 - 6934.268888039822) / 6319.656911553861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.219395061728395 > 10.317752418251343.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -97.90661123247455 (T) = (0 -19398.400938317056) / Math.Sqrt((5702.81802474074 / (23)) + (138619.4751749903 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.9671745465638503 = (9861.047141038445 - 19398.400938317056) / 9861.047141038445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.212590607940445 > 10.655611694346437.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -101.22812625307964 (T) = (0 -20605.594438205466) / Math.Sqrt((4288.942037490409 / (23)) + (155574.62669505473 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0193384716505758 = (10204.131069400551 - 20605.594438205466) / 10204.131069400551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchFinally(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.036680939283645 > 2.9930731329231977.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -231.37047833704682 (T) = (0 -9056.262241327882) / Math.Sqrt((435.7367036726216 / (23)) + (10498.342209990305 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.174330683246488 = (2852.9674898475787 - 9056.262241327882) / 2852.9674898475787 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionSoftware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.025517317366585 > 17.344051615709844.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -196.96437530937 (T) = (0 -40507.40498065106) / Math.Sqrt((17459.14004213164 / (23)) + (209677.97859402347 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.4407636104583594 = (16596.201617838786 - 40507.40498065106) / 16596.201617838786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.MultipleNestedTryCatch_LastCatches(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.172745436217008 > 3.728499906612711.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -201.23342857195541 (T) = (0 -9260.220726006284) / Math.Sqrt((1454.135891843331 / (23)) + (11107.655563727167 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.6046877456125872 = (3555.2133807994733 - 9260.220726006284) / 3555.2133807994733 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: Hardware)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.427352498943364 > 13.707230837195409.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -55.97756414895842 (T) = (0 -18097.38758820882) / Math.Sqrt((11466.47840815323 / (23)) + (115282.42358598115 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.38854567933626977 = (13033.33974353616 - 18097.38758820882) / 13033.33974353616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.40559239530155 > 4.013775014729449.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -195.7955992785137 (T) = (0 -9255.100759161543) / Math.Sqrt((1866.153113912687 / (23)) + (10304.74444559318 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.4173401718360013 = (3828.629858135429 - 9255.100759161543) / 3828.629858135429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Exceptions.Handling.TryAndCatchDeep_CaugtAtTheTop(kind: Software)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.360349493969714 > 13.03726182826291.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -89.17416194335344 (T) = (0 -20504.711661704143) / Math.Sqrt((9340.46970564664 / (23)) + (117722.71761808921 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.653210975302468 = (12402.961248156875 - 20504.711661704143) / 12402.961248156875 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-05-16T18:44:11Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.CastingPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ObjObjrefValueType - Duration of single invocation	87.03 μs	237.17 μs	2.73	0.06	True
IFooObjIsIFooInterAlia - Duration of single invocation	239.74 μs	505.30 μs	2.11	0.06	True
ObjObjIsFoo - Duration of single invocation	177.96 μs	505.93 μs	2.84	0.07	True
IntObj - Duration of single invocation	193.13 μs	286.32 μs	1.48	0.11	True
FooObjIsFoo - Duration of single invocation	193.17 μs	258.02 μs	1.34	0.07	True
FooObjIsFoo2 - Duration of single invocation	193.17 μs	257.20 μs	1.33	0.06	True
ObjScalarValueType - Duration of single invocation	127.82 μs	204.67 μs	1.60	0.07	True
IFooFooIsIFoo - Duration of single invocation	87.08 μs	204.67 μs	2.35	0.12	True
FooObjIsNull - Duration of single invocation	90.09 μs	205.06 μs	2.28	0.07	True
ScalarValueTypeObj - Duration of single invocation	193.21 μs	285.78 μs	1.48	0.09	True
ObjrefValueTypeObj - Duration of single invocation	193.19 μs	257.79 μs	1.33	0.05	True
CheckArrayIsNonvariantGenericInterface - Duration of single invocation	1.63 ns	4.31 ns	2.64	0.17	True
FooObjIsDescendant - Duration of single invocation	193.17 μs	476.66 μs	2.47	0.07	True
ObjFooIsObj2 - Duration of single invocation	87.01 μs	205.00 μs	2.36	0.12	True
ObjFooIsObj - Duration of single invocation	87.63 μs	237.03 μs	2.70	0.06	True
IFooObjIsIFoo - Duration of single invocation	240.03 μs	505.61 μs	2.11	0.06	True
ObjInt - Duration of single invocation	108.79 μs	204.57 μs	1.88	0.08	True
IFooObjIsDescendantOfIFoo - Duration of single invocation	239.91 μs	504.90 μs	2.10	0.04	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.ObjObjrefValueType

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 237.16736590909088 > 91.3755762069311.
IsChangePoint: Marked as a change because one of 4/3/2023 12:37:39 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -31.279619089289795 (T) = (0 -222162.47845622708) / Math.Sqrt((3933.136222590503 / (23)) + (279915330.8355098 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.5524506268313143 = (87038.89357187135 - 222162.47845622708) / 87038.89357187135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.IFooObjIsIFooInterAlia

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.3018051555299 > 251.91114147727274.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -22.046620615448884 (T) = (0 -512249.19777168933) / Math.Sqrt((207400424.36797905 / (23)) + (2104108533.597014 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.1091241064840955 = (242872.9519504698 - 512249.19777168933) / 242872.9519504698 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjObjIsFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.9328427083334 > 186.854410337358.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -86.19439125213313 (T) = (0 -496310.13834111084) / Math.Sqrt((26861643.034419917 / (23)) + (185663058.9834537 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.7714554722522016 = (179079.23952239734 - 496310.13834111084) / 179079.23952239734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.IntObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 286.3198097402597 > 202.82995143325613.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -19.965676300770205 (T) = (0 -268644.55412138987) / Math.Sqrt((21344.783517156044 / (23)) + (214121892.55303743 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3904436968473379 = (193207.79024027276 - 268644.55412138987) / 193207.79024027276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.FooObjIsFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.02227137978144 > 202.83438907996384.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -18.478559813149165 (T) = (0 -268951.7819138302) / Math.Sqrt((65744968.27676709 / (23)) + (196979596.63998896 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3788184501394291 = (195059.6047555305 - 268951.7819138302) / 195059.6047555305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.FooObjIsFoo2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 257.19925722446243 > 202.84630683256177.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -12.880702677626868 (T) = (0 -286268.9278818267) / Math.Sqrt((15828.677375097579 / (23)) + (783462622.2831485 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.48188962169841004 = (193178.30673092286 - 286268.9278818267) / 193178.30673092286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjScalarValueType

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 204.67422367424243 > 80.36192273780489.
IsChangePoint: Marked as a change because one of 3/13/2023 12:04:12 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -29.38182321336266 (T) = (0 -215478.2623803219) / Math.Sqrt((114452295.84531285 / (23)) + (250204102.880373 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.736387068138849 = (78745.53453685163 - 215478.2623803219) / 78745.53453685163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 204.66749391233768 > 91.3880819626736.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -31.651411619589528 (T) = (0 -215619.6555519144) / Math.Sqrt((3826.211865390729 / (23)) + (247567373.79723462 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.477450892534846 = (87032.86761470354 - 215619.6555519144) / 87032.86761470354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.FooObjIsNull

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.05504715909083 > 93.36879139640118.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -30.598494065906017 (T) = (0 -220010.1775945521) / Math.Sqrt((2966722.6617880054 / (23)) + (272513407.2233178 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.4685087357508817 = (89126.75673705018 - 220010.1775945521) / 89126.75673705018 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ScalarValueTypeObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 285.7767196212122 > 202.8359325289352.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -9.576422914801219 (T) = (0 -272559.7706370058) / Math.Sqrt((963243239.3230557 / (23)) + (223954264.72421053 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.36021652073119464 = (200379.69432284887 - 272559.7706370058) / 200379.69432284887 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjrefValueTypeObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 257.79433230874315 > 202.84398292601495.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -24.887437933934134 (T) = (0 -262002.42963033455) / Math.Sqrt((261284.0251015692 / (23)) + (106448323.57575072 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -0.3552386809280051 = (193325.67267850364 - 262002.42963033455) / 193325.67267850364 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.CheckArrayIsNonvariantGenericInterface

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.305957841225985 > 1.6954771021302304.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -14.654968331809338 (T) = (0 -4.642027571995341) / Math.Sqrt((0.0525843157824081 / (23)) + (0.6002134783798087 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.8514952820332538 = (1.6279274951790739 - 4.642027571995341) / 1.6279274951790739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.FooObjIsDescendant

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 476.65719646464646 > 202.82147247032287.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -104.19760192203908 (T) = (0 -499680.56037764123) / Math.Sqrt((16995.76546262236 / (23)) + (129761302.35889682 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.5863412053683754 = (193199.78328477012 - 499680.56037764123) / 193199.78328477012 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjFooIsObj2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 204.9974430465368 > 91.3680738576389.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -35.94693432387962 (T) = (0 -211322.74680950693) / Math.Sqrt((2361.8886134439026 / (23)) + (179324802.4484732 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.4280998176524382 = (87032.14969713242 - 211322.74680950693) / 87032.14969713242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjFooIsObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 237.02980018939397 > 91.18070474986506.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -31.008059953428216 (T) = (0 -219762.6893929301) / Math.Sqrt((856436.9728425711 / (23)) + (275462045.8312766 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.5333468050709562 = (86747.96871594325 - 219762.6893929301) / 86747.96871594325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.IFooObjIsIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.60788181089737 > 251.93641597064394.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -42.468478783247924 (T) = (0 -502263.2950182842) / Math.Sqrt((50813.01686725349 / (23)) + (572714381.5142883 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0941614184085169 = (239839.81874710746 - 502263.2950182842) / 239839.81874710746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.ObjInt

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 204.56638862476805 > 114.22448912109374.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -25.882991602579278 (T) = (0 -219662.24326327976) / Math.Sqrt((8383.361118184326 / (23)) + (275149436.23581624 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0188303844935773 = (108806.68576740385 - 219662.24326327976) / 108806.68576740385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf.IFooObjIsDescendantOfIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 504.90265638950893 > 251.90016582623113.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -107.53480202824441 (T) = (0 -501591.4757249408) / Math.Sqrt((19392.929778286696 / (23)) + (88885799.08098133 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0916812577460564 = (239803.01676816825 - 501591.4757249408) / 239803.01676816825 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Reflection.Invoke

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Property_Get_class - Duration of single invocation	13.31 ns	16.10 ns	1.21	0.05	False
Property_Get_int - Duration of single invocation	15.55 ns	19.61 ns	1.26	0.17	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Reflection.Invoke*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Reflection.Invoke* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Reflection.Invoke*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Reflection.Invoke* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Reflection.Invoke.Property_Get_class

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.100017506641088 > 13.996522136281254.
IsChangePoint: Marked as a change because one of 3/8/2023 9:36:36 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -34.55118121688192 (T) = (0 -15.779150575628218) / Math.Sqrt((0.09844238728346708 / (23)) + (0.010019904898136206 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.18207158798074838 = (13.348726706630904 - 15.779150575628218) / 13.348726706630904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Invoke.Property_Get_int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.61272630597015 > 16.822158604809196.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -29.299117456308984 (T) = (0 -19.44158200753337) / Math.Sqrt((0.14989742378122908 / (23)) + (0.1189758182185113 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.22122706710763737 = (15.919711027678863 - 19.44158200753337) / 15.919711027678863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-05-16T18:44:12Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
IFooFooIsIFoo - Duration of single invocation	217.37 μs	412.51 μs	1.90	0.04	True
FooObjIsDescendant - Duration of single invocation	152.30 μs	494.29 μs	3.25	0.06	True
FooObjIsNull - Duration of single invocation	217.51 μs	304.53 μs	1.40	0.06	False
ObjrefValueTypeObj - Duration of single invocation	565.73 μs	901.98 μs	1.59	0.18	True
IFooObjIsIFooInterAlia - Duration of single invocation	174.00 μs	390.44 μs	2.24	0.08	True
ObjFooIsObj - Duration of single invocation	217.56 μs	412.99 μs	1.90	0.05	True
IFooObjIsIFoo - Duration of single invocation	195.92 μs	489.80 μs	2.50	0.06	True
FooObjIsFoo - Duration of single invocation	304.52 μs	519.40 μs	1.71	0.06	True
FooObjCastIfIsa - Duration of single invocation	246.37 μs	272.32 μs	1.11	0.03	True
IFooObjIsDescendantOfIFoo - Duration of single invocation	195.82 μs	465.88 μs	2.38	0.06	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 412.5123285087719 > 228.45196133029523.
IsChangePoint: Marked as a change because one of 4/27/2023 1:43:27 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -81.35887568498316 (T) = (0 -417740.5475754906) / Math.Sqrt((10222.469987015173 / (23)) + (90817508.4708323 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.9202717949391725 = (217542.4065887106 - 417740.5475754906) / 217542.4065887106 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsDescendant

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 494.28714726562504 > 159.89940530323284.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -125.83623683296922 (T) = (0 -486995.3429181032) / Math.Sqrt((18278.368002331976 / (23)) + (106098339.03416128 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.197429503142118 = (152308.39098705142 - 486995.3429181032) / 152308.39098705142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 304.5273930288461 > 228.36619007161457.
IsChangePoint: Marked as a change because one of 4/27/2023 1:43:27 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -18.190225365099774 (T) = (0 -291719.81203510327) / Math.Sqrt((118697213.41361828 / (23)) + (156811797.25889605 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3269645189444968 = (219839.94889866764 - 291719.81203510327) / 219839.94889866764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.ObjrefValueTypeObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 901.9774192982458 > 593.9717681392609.
IsChangePoint: Marked as a change because one of 4/27/2023 8:46:44 AM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -9.41160320901919 (T) = (0 -779297.5104420275) / Math.Sqrt((2951431640.2877316 / (23)) + (4953172335.965967 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3488189138219712 = (577762.8875575553 - 779297.5104420275) / 577762.8875575553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 390.44204796747965 > 182.74184475713955.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -22.7586474116937 (T) = (0 -364800.09532957815) / Math.Sqrt((15285959.794839712 / (23)) + (1035224318.5815125 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.0866715855347742 = (174823.91472546305 - 364800.09532957815) / 174823.91472546305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 412.9900117324562 > 228.43292746644633.
IsChangePoint: Marked as a change because one of 4/27/2023 1:43:27 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -92.5280787779277 (T) = (0 -416231.8272922029) / Math.Sqrt((283396.4408655323 / (23)) + (68893741.39127181 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.9122302428845778 = (217668.25874708575 - 416231.8272922029) / 217668.25874708575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 489.798603515625 > 205.574407443928.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -81.34099850398036 (T) = (0 -475188.8690374888) / Math.Sqrt((35901.79454006954 / (23)) + (165061011.34676287 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -1.4260118819330196 = (195872.44092921077 - 475188.8690374888) / 195872.44092921077 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 519.3963965053763 > 319.74783381610575.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -257.10638952012937 (T) = (0 -517195.0789153472) / Math.Sqrt((82698.29682600222 / (23)) + (10200028.288868053 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.697838932859326 = (304619.63670743513 - 517195.0789153472) / 304619.63670743513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.FooObjCastIfIsa

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 272.31966979679805 > 258.61077813683147.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -21.80744225801384 (T) = (0 -275230.3639970163) / Math.Sqrt((32136.84503000472 / (23)) + (26284864.16558563 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.1172277465714045 = (246351.17131816212 - 275230.3639970163) / 246351.17131816212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 465.8790017156864 > 205.61282708007815.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -74.64023417348626 (T) = (0 -474586.7527794164) / Math.Sqrt((67720378.34293747 / (23)) + (162564921.30244464 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.4030777043191873 = (197491.2221633178 - 474586.7527794164) / 197491.2221633178 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.CtorDefaultSize<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Queue - Duration of single invocation	12.68 ns	15.55 ns	1.23	0.11	False
Stack - Duration of single invocation	11.97 ns	14.71 ns	1.23	0.15	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorDefaultSize&lt;String&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.CtorDefaultSize&lt;String&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Collections.CtorDefaultSize<String>.Queue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.553641708731483 > 13.378297398112277.
IsChangePoint: Marked as a change because one of 3/9/2023 1:54:27 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -5.68460024373312 (T) = (0 -14.560907495495456) / Math.Sqrt((0.09410953346861368 / (23)) + (1.157511162414781 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.1252236704380474 = (12.940456087123481 - 14.560907495495456) / 12.940456087123481 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CtorDefaultSize<String>.Stack

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.705204589020115 > 12.391569354127695.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -8.026558269682514 (T) = (0 -14.280442658330609) / Math.Sqrt((0.0596233403718168 / (23)) + (1.2297622329509756 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.1954006226012879 = (11.946156282949909 - 14.280442658330609) / 11.946156282949909 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DivideBenchmark - Duration of single invocation	17.50 ns	30.37 ns	1.74	0.44	False
DivisionOperatorBenchmark - Duration of single invocation	18.80 ns	22.80 ns	1.21	0.40	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.3723122367665 > 19.563131816432175.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -10.828606440172406 (T) = (0 -23.58323245641052) / Math.Sqrt((0.4149315010717872 / (23)) + (3.5733753411967166 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.30283289124556745 = (18.10150220713562 - 23.58323245641052) / 18.10150220713562 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Int64>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.800271363886427 > 19.74537870813019.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -33.337446366347365 (T) = (0 -22.942157561495556) / Math.Sqrt((0.41918010247955517 / (23)) + (0.04035156529365353 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.26606395121346654 = (18.120852062414784 - 22.942157561495556) / 18.120852062414784 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-05-16T18:44:13Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Diagnostics.Perf_Activity

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EnumerateActivityTagObjectsLarge - Duration of single invocation	6.26 μs	9.16 μs	1.46	0.30	True
EnumerateActivityEventsLarge - Duration of single invocation	7.88 μs	12.40 μs	1.57	0.16	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Diagnostics.Perf_Activity*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Diagnostics.Perf_Activity* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Diagnostics.Perf_Activity*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Diagnostics.Perf_Activity* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Diagnostics.Perf_Activity.EnumerateActivityTagObjectsLarge

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.156447261206536 > 6.782858200460722.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -27.27698749326993 (T) = (0 -9062.682446749923) / Math.Sqrt((23526.50863570092 / (23)) + (112198.74611006468 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.3841538564823621 = (6547.453091509272 - 9062.682446749923) / 6547.453091509272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Diagnostics.Perf_Activity.EnumerateActivityEventsLarge

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.399536302415074 > 8.130309300742493.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -38.576209293246116 (T) = (0 -12227.30997411932) / Math.Sqrt((285242.0802713441 / (23)) + (50772.60623226001 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.6567269265663399 = (7380.401548407926 - 12227.30997411932) / 7380.401548407926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in HardwareIntrinsics.RayTracer.SoA

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Render - Duration of single invocation	0.69 ns	5.28 ms	7593574.55	0.07	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'HardwareIntrinsics.RayTracer.SoA*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter HardwareIntrinsics.RayTracer.SoA* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'HardwareIntrinsics.RayTracer.SoA*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter HardwareIntrinsics.RayTracer.SoA* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

HardwareIntrinsics.RayTracer.SoA.Render

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.277020531914893 > 0.7296829861689195.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -173.06225753047406 (T) = (0 -5348666.2715910515) / Math.Sqrt((0.05000001262228834 / (23)) + (14327731148.849329 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -8019890.285161609 = (0.6669250344436896 - 5348666.2715910515) / 0.6669250344436896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DivisionOperatorBenchmark - Duration of single invocation	52.05 ns	55.41 ns	1.06	0.44	False
DivideBenchmark - Duration of single invocation	48.07 ns	55.31 ns	1.15	0.43	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.41084198000168 > 50.60946161897349.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -15.999294244357229 (T) = (0 -55.62700798564247) / Math.Sqrt((3.4286995733268806 / (23)) + (0.07183015332523568 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.12716550062731793 = (49.35123365174285 - 55.62700798564247) / 49.35123365174285 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<Byte>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.310677969253355 > 50.71699424139733.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -13.627217716864768 (T) = (0 -55.53841516623031) / Math.Sqrt((4.002392350046962 / (23)) + (0.06908171174241483 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.11569877398647237 = (49.77904113651352 - 55.53841516623031) / 49.77904113651352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Threading.Tests.Perf_ThreadStatic

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SetThreadStatic - Duration of single invocation	2.18 ns	5.93 ns	2.71	0.12	True
GetThreadStatic - Duration of single invocation	2.45 ns	4.39 ns	1.80	0.20	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Threading.Tests.Perf_ThreadStatic*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Threading.Tests.Perf_ThreadStatic* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Threading.Tests.Perf_ThreadStatic*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Threading.Tests.Perf_ThreadStatic* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Threading.Tests.Perf_ThreadStatic.SetThreadStatic

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.925997766129071 > 2.2971045201762186.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -123.07032834517231 (T) = (0 -5.920085801049014) / Math.Sqrt((0.011338934063228508 / (23)) + (0.00654931579856416 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -1.7310249940682239 = (2.1677157162264784 - 5.920085801049014) / 2.1677157162264784 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Threading.Tests.Perf_ThreadStatic.GetThreadStatic

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.392452616466371 > 2.573845245754667.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -19.193796966859637 (T) = (0 -4.4926016159320605) / Math.Sqrt((0.0323304140417721 / (23)) + (0.13492140644521142 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.7721104626904849 = (2.5351701886073306 - 4.4926016159320605) / 2.5351701886073306 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.MathBenchmarks.Double

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Sqrt - Duration of single invocation	9.96 μs	19.68 μs	1.98	0.02	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Double*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Double*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Double* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.Sqrt

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.681977820633918 > 10.455954834406967.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -444.51688010874295 (T) = (0 -19680.52868849281) / Math.Sqrt((10625.127053469838 / (23)) + (138.6949044668274 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.9619214593579308 = (10031.252063950393 - 19680.52868849281) / 10031.252063950393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-05-16T18:44:15Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.DelegatePerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
MulticastDelegateInvoke - Duration of single invocation	2.04 ms	2.67 ms	1.31	0.50	False
MulticastDelegateInvoke - Duration of single invocation	21.05 ms	29.79 ms	1.42	0.46	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.DelegatePerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.DelegatePerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.DelegatePerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.DelegatePerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6730709404761908 > 2.1392912355468754.
IsChangePoint: Marked as a change because one of 5/2/2023 12:34:35 AM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -8.482737328582363 (T) = (0 -2692963.200595972) / Math.Sqrt((48827807335.94379 / (23)) + (3025440010.699104 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.17907029002735278 = (2283971.72193483 - 2692963.200595972) / 2283971.72193483 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 1000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.785631035714278 > 22.020101160256413.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -11.807317789549161 (T) = (0 -27555466.15545351) / Math.Sqrt((2887574609520.1797 / (23)) + (428134588846.31824 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.20221394897909178 = (22920600.92868939 - 27555466.15545351) / 22920600.92868939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Object

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetType_ - Duration of single invocation	0.29 ns	1.48 ns	5.08	0.38	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Object*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Object* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Object*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Object* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Object.GetType_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4824744676381365 > 0.30310485238226764.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -23.78306728708664 (T) = (0 -1.5649746482112314) / Math.Sqrt((0.004143219221617132 / (23)) + (0.04197714717286738 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -4.861915277912283 = (0.2669732628357938 - 1.5649746482112314) / 0.2669732628357938 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetHashCodeBenchmark - Duration of single invocation	100.74 ns	106.67 ns	1.06	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.GetHashCodeBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.66889636117192 > 104.97681493388235.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/22/2023 7:51:01 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -17.64203541266722 (T) = (0 -106.78908447144413) / Math.Sqrt((2.4131031634583295 / (23)) + (0.03130269800868159 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.05712815997207252 = (101.0181059544051 - 106.78908447144413) / 101.0181059544051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Threading.Tests.Perf_EventWaitHandle

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Set_Reset - Duration of single invocation	260.20 ns	288.09 ns	1.11	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Threading.Tests.Perf_EventWaitHandle*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Threading.Tests.Perf_EventWaitHandle* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Threading.Tests.Perf_EventWaitHandle*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Threading.Tests.Perf_EventWaitHandle* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Threading.Tests.Perf_EventWaitHandle.Set_Reset

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 288.0913367749874 > 275.6449664688903.
IsChangePoint: Marked as a change because one of 4/1/2023 8:54:51 PM, 4/3/2023 12:37:39 PM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -13.923984933238486 (T) = (0 -287.05849887500506) / Math.Sqrt((17.903274654324402 / (23)) + (28.88570273654259 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.0866775333101183 = (264.1616211578414 - 287.05849887500506) / 264.1616211578414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DivideBenchmark - Duration of single invocation	16.65 ns	19.45 ns	1.17	0.02	True
DivisionOperatorBenchmark - Duration of single invocation	16.67 ns	18.66 ns	1.12	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.452509820701124 > 17.5527612755145.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -30.814204546797274 (T) = (0 -19.10455613953969) / Math.Sqrt((0.021637211109554747 / (23)) + (0.07549415180445819 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.14241633982671034 = (16.72293670312664 - 19.10455613953969) / 16.72293670312664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Numerics.Tests.Perf_VectorOf<UInt64>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.65890442354529 > 17.536766728432475.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/11/2023 5:02:01 PM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -3.2222866966930006 (T) = (0 -19.37813716477257) / Math.Sqrt((3.736703826366623 / (26)) + (3.2668872485104328 / (12))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (12) - 2, .025) and -0.12012841359503072 = (17.299924660047513 - 19.37813716477257) / 17.299924660047513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-05-16T18:44:16Z

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Environment

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
SetEnvironmentVariable - Duration of single invocation	151.10 ns	535.63 ns	3.54	0.07	True
GetFolderPath - Duration of single invocation	193.14 ns	274.17 ns	1.42	0.22	False
GetEnvironmentVariables - Duration of single invocation	9.25 μs	12.51 μs	1.35	0.52	False
GetEnvironmentVariable - Duration of single invocation	104.57 ns	316.49 ns	3.03	0.07	True
ExpandEnvironmentVariables - Duration of single invocation	330.50 ns	361.34 ns	1.09	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Environment*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Environment* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Environment*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Environment* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.SetEnvironmentVariable

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 535.6308477213498 > 159.6863476215757.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -187.53727226914015 (T) = (0 -529.1645795996607) / Math.Sqrt((39.477891224605315 / (23)) + (34.46365269869624 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.4488112621927507 = (153.43390500969844 - 529.1645795996607) / 153.43390500969844 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Environment.GetFolderPath(folder: System, option: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 274.17357799555464 > 201.9586515977388.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -13.948075559549286 (T) = (0 -256.4388206590076) / Math.Sqrt((8.451634350675443 / (23)) + (317.9700527486986 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.337947267945024 = (191.66586516737416 - 256.4388206590076) / 191.66586516737416 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Environment.GetEnvironmentVariables

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.512899718976199 > 8.645290183164445.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -9.862407642902973 (T) = (0 -11638.798131917707) / Math.Sqrt((890712.5416345999 / (23)) + (914489.4011427796 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.36528240265866835 = (8524.828349983136 - 11638.798131917707) / 8524.828349983136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Environment.GetEnvironmentVariable

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 316.48862461943435 > 109.31799181108694.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -71.06025145170587 (T) = (0 -317.0862230962784) / Math.Sqrt((4.938774330139719 / (23)) + (131.5324478452033 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -2.0459691906539605 = (104.10027260590935 - 317.0862230962784) / 104.10027260590935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Environment.ExpandEnvironmentVariables

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 361.340410936286 > 346.9124940774376.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 3/29/2023 4:47:17 AM, 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -32.76739555162619 (T) = (0 -359.3305433458296) / Math.Sqrt((6.4608377288254895 / (23)) + (7.139000563121051 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.08616813687587115 = (330.82405121859546 - 359.3305433458296) / 330.82405121859546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in BilinearTest

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Interpol_AVX - Duration of single invocation	0.27 ns	2.07 μs	7678.84	0.20	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BilinearTest*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter BilinearTest* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BilinearTest*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter BilinearTest* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

BilinearTest.Interpol_AVX

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.070280368865828 > 0.276973228378407.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -461.9816546208022 (T) = (0 -2074.4793648522927) / Math.Sqrt((0.002983502817394357 / (23)) + (302.3789047811976 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -8212.859556904445 = (0.25255841672000795 - 2074.4793648522927) / 0.25255841672000795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.LowLevelPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GenericClassWithSTringGenericInstanceMethod - Duration of single invocation	87.04 μs	180.74 μs	2.08	0.06	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.7417880268199 > 91.38577279560631.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -19.88839492621346 (T) = (0 -173593.55464786605) / Math.Sqrt((60238121.983219214 / (23)) + (230459528.9133019 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.9449374430050269 = (89254.05558528156 - 173593.55464786605) / 89254.05558528156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Compare	a23e797b61acfa9ae5e0575240f464cf62f2d83b
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.EnumPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ObjectGetTypeNoBoxing - Duration of single invocation	0.28 ns	1.52 ns	5.47	0.39	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.EnumPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.EnumPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.EnumPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.EnumPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.EnumPerf.ObjectGetTypeNoBoxing

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5176319864446732 > 0.29667175534007.
IsChangePoint: Marked as a change because one of 5/11/2023 4:41:38 AM, 5/15/2023 6:46:12 PM falls between 5/7/2023 5:11:33 AM and 5/15/2023 6:46:12 PM.
IsRegressionStdDev: Marked as regression because -14.980654576716333 (T) = (0 -1.5589376571800906) / Math.Sqrt((0.09163184580950955 / (23)) + (0.032273117268904625 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -3.0438509286383186 = (0.38550819124903546 - 1.5589376571800906) / 0.38550819124903546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

LeVladIonescu · 2023-07-26T07:34:15Z

Not relevant, will look at new ones after mono run fix.

performanceautofiler bot added arch-x64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono untriaged labels May 16, 2023

kotlarmilos mentioned this issue May 17, 2023

[Perf] Linux/arm64: 3189 Improvements on 5/11/2023 10:01:37 AM #17878

Closed

kotlarmilos removed the untriaged label May 17, 2023

kotlarmilos mentioned this issue May 17, 2023

.NET 8 per-preview performance overview dotnet/runtime#77490

Closed

kotlarmilos added the LinkedFromMaster Is this issue linked from the master perf issue label May 17, 2023

This was referenced May 17, 2023

[Perf] Linux/arm64: 2 Regressions on 5/11/2023 7:02:02 PM #17871

Closed

[Perf] Linux/x64: 1 Regression on 5/11/2023 1:19:27 PM #17875

Closed

LeVladIonescu added the Look Again label Jul 12, 2023

LeVladIonescu removed the Look Again label Jul 26, 2023

LeVladIonescu closed this as completed Jul 26, 2023

[Perf] Linux/x64: 100 Regressions on 5/11/2023 9:33:40 AM #17874

[Perf] Linux/x64: 100 Regressions on 5/11/2023 9:33:40 AM #17874

Comments

performanceautofiler bot commented May 16, 2023 • edited Loading

Run Information

Regressions in System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_SameReference

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_Equal

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_AllBitsDifferent

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_CascadingErrors

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_FirstBitDifferent

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_VersusZero

Description of detection logic

JIT Disasms

System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.FixedTimeEquals_256Bit_LastBitDifferent

Description of detection logic

JIT Disasms

Docs

Run Information

Regressions in System.Xml.Linq.Perf_XElementList

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Xml.Linq.Perf_XElementList.Enumerator

Description of detection logic

JIT Disasms

Docs

performanceautofiler bot commented May 16, 2023

Run Information

Regressions in Exceptions.Handling

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

Exceptions.Handling.MultipleNestedTryCatch_FirstCatches(kind: Hardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhen(kind: Hardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.MultipleNestedTryFinally(kind: Hardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Hardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.CatchAndThrowOtherDeep(kind: Hardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionHardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionHardware)

Description of detection logic

JIT Disasms

Exceptions.Handling.MultipleNestedTryFinally(kind: Software)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Software)

Description of detection logic

JIT Disasms

Exceptions.Handling.ThrowAndCatchDeep(kind: Hardware)

performanceautofiler bot commented May 16, 2023 •

edited

Loading