Closed
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 2b0669b3fc36cafa09333e78f79525d96d0ed9c7 |
| Compare | 19d545c62c3eea1836e8c3830eb654e2a022cac7 |
| Diff | Diff |
| Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Improvements in System.Buffers.Text.Tests.Utf8FormatterTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| FormatterUInt32 - Duration of single invocation | 211.09 ns | 150.77 ns | 0.71 | 0.19 | False | |||||
| FormatterInt32 - Duration of single invocation | 164.76 ns | 109.56 ns | 0.66 | 0.30 | False | |||||
| FormatterInt32 - Duration of single invocation | 316.26 ns | 220.92 ns | 0.70 | 0.19 | False | |||||
| FormatterInt32 - Duration of single invocation | 214.39 ns | 161.55 ns | 0.75 | 0.11 | False | |||||
| FormatterUInt64 - Duration of single invocation | 385.76 ns | 303.17 ns | 0.79 | 0.08 | False | |||||
| FormatterInt64 - Duration of single invocation | 449.62 ns | 334.95 ns | 0.74 | 0.07 | False | |||||
| FormatterInt64 - Duration of single invocation | 370.81 ns | 297.51 ns | 0.80 | 0.06 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifactsor build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0 - CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0 - Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release - Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packagesWindows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packagesPayloads
Histogram
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 150.77460503073985 < 201.4553571240788.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 49.46039352711033 (T) = (0 -150.134052661451) / Math.Sqrt((340.3646597169272 / (299)) + (22.01889276162699 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.30806659987071255 = (216.9776059854872 - 150.134052661451) / 216.9776059854872 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 109.56467104087169 < 158.71995633383213.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 46.97543663339671 (T) = (0 -104.10191657397577) / Math.Sqrt((402.4031587417159 / (299)) + (6.975872932547617 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.36073444763639173 = (162.84612269356816 - 104.10191657397577) / 162.84612269356816 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 220.92192874270145 < 305.3859741286148.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 10.623469901257128 (T) = (0 -213.74281453602745) / Math.Sqrt((1415.9038724335117 / (299)) + (15.480719516286978 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.10196932536459741 = (238.01282135803024 - 213.74281453602745) / 238.01282135803024 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 161.54821574636404 < 203.90813924071867.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 61.87819963709965 (T) = (0 -152.20149883484825) / Math.Sqrt((317.4473998956504 / (299)) + (9.637256227850742 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.32263411918597584 = (224.69614007121257 - 152.20149883484825) / 224.69614007121257 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 18446744073709551615)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 303.1671153651589 < 349.9771486536805.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 32.80004876072968 (T) = (0 -303.6373983358373) / Math.Sqrt((373.8245379319783 / (299)) + (58.49268248318182 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.15932747153498658 = (361.1839188919968 - 303.6373983358373) / 361.1839188919968 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 334.95399810174644 < 431.8086743326261.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 9.534252760339916 (T) = (0 -342.56390007829066) / Math.Sqrt((1571.984315909507 / (299)) + (24.948337593039366 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.06413228718102615 = (366.03880589750963 - 342.56390007829066) / 366.03880589750963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 297.50632629577746 < 348.2067924086597.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 33.041973015412104 (T) = (0 -298.17266593934926) / Math.Sqrt((429.43311644157575 / (299)) + (53.86621539214488 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.16368662330577763 = (356.53222135219875 - 298.17266593934926) / 356.53222135219875 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 2b0669b3fc36cafa09333e78f79525d96d0ed9c7 |
| Compare | 19d545c62c3eea1836e8c3830eb654e2a022cac7 |
| Diff | Diff |
| Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Improvements in System.Text.Json.Tests.Perf_Basic
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteBasicUtf16 - Duration of single invocation | 33.34 ms | 23.63 ms | 0.71 | 0.14 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 30.52 ms | 21.93 ms | 0.72 | 0.11 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 40.89 ms | 31.28 ms | 0.76 | 0.10 | False | |||||
| WriteBasicUtf8 - Duration of single invocation | 30.28 ms | 21.35 ms | 0.71 | 0.14 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 38.15 ms | 29.26 ms | 0.77 | 0.12 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 17.35 μs | 14.64 μs | 0.84 | 0.02 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 19.70 μs | 16.78 μs | 0.85 | 0.01 | False | |||||
| WriteBasicUtf8 - Duration of single invocation | 32.02 ms | 23.25 ms | 0.73 | 0.12 | False | |||||
| WriteBasicUtf8 - Duration of single invocation | 40.31 ms | 30.89 ms | 0.77 | 0.09 | False | |||||
| WriteBasicUtf8 - Duration of single invocation | 38.48 ms | 30.97 ms | 0.80 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifactsor build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0 - CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0 - Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release - Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Basic*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packagesWindows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Basic*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packagesPayloads
Histogram
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 23.628659853846155 < 31.600656197338598.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 16.023337806833208 (T) = (0 -23402075.81366618) / Math.Sqrt((9608732838823.355 / (299)) + (257210668215.94067 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.1206758799378946 = (26613708.506043624 - 23402075.81366618) / 26613708.506043624 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 21.929609008333337 < 29.22715269071085.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 16.645193872584972 (T) = (0 -21889713.377135623) / Math.Sqrt((9731440329228.646 / (299)) + (211589898450.18973 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.13079358262884186 = (25183561.62548733 - 21889713.377135623) / 25183561.62548733 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 31.28160258653846 < 38.71062038628663.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 15.195539934643962 (T) = (0 -31226301.878604334) / Math.Sqrt((10432432724766.434 / (299)) + (378118028662.23285 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.0951610253182409 = (34510341.345085114 - 31226301.878604334) / 34510341.345085114 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 21.348007867132864 < 28.97564741280187.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 17.478112261377948 (T) = (0 -21861805.359142415) / Math.Sqrt((9608946930657.885 / (299)) + (162001983175.1209 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.13359709113454887 = (25232839.289252046 - 21861805.359142415) / 25232839.289252046 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 29.257652076923076 < 37.29896955357142.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 16.217215114593603 (T) = (0 -29676983.721373927) / Math.Sqrt((9951549895491.605 / (299)) + (323747408073.64386 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.10255046303562507 = (33068136.42330954 - 29676983.721373927) / 33068136.42330954 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.636445935921715 < 16.54203415027221.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/15/2023 5:33:41 AM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 13.199458653764697 (T) = (0 -14823.014783987393) / Math.Sqrt((1194055.5635173083 / (299)) + (44722.7665818584 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.06136945722740455 = (15792.171795520408 - 14823.014783987393) / 15792.171795520408 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.778523349655423 < 18.683992698837496.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/15/2023 5:33:41 AM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 11.204409249134347 (T) = (0 -17028.533388157644) / Math.Sqrt((1441659.1871793026 / (299)) + (52291.63256482203 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.050212339277003165 = (17928.77933915797 - 17028.533388157644) / 17928.77933915797 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 23.24726492727273 < 30.557662402901784.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 15.580593370239576 (T) = (0 -23475120.000298344) / Math.Sqrt((9370614116511.25 / (299)) + (177453078746.80173 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.113057189857381 = (26467456.223613314 - 23475120.000298344) / 26467456.223613314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 30.891775875 < 38.8816101340873.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 14.178221895908628 (T) = (0 -31320260.680333126) / Math.Sqrt((10344153389267.945 / (299)) + (551298445763.3926 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.09342541137686476 = (34547913.73305635 - 31320260.680333126) / 34547913.73305635 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 30.96725975892857 < 37.939109605238095.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 3/14/2023 5:06:59 PM, 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM falls between 4/23/2023 8:38:14 PM and 5/2/2023 3:18:41 PM.
IsImprovementStdDev: Marked as improvement because 12.76713326356154 (T) = (0 -29997060.177698057) / Math.Sqrt((9646676956062.34 / (299)) + (863122879354.529 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.09386417106927883 = (33104374.88504992 - 29997060.177698057) / 33104374.88504992 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository