Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 38 Regressions on 4/9/2023 11:24:15 AM #15636

Open
performanceautofiler bot opened this issue Apr 11, 2023 · 3 comments
Open

[Perf] Linux/x64: 38 Regressions on 4/9/2023 11:24:15 AM #15636

performanceautofiler bot opened this issue Apr 11, 2023 · 3 comments
Assignees
@stephentoub
Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression untriaged wasm-aot

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 11, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 854.43 μs 1.10 ms 1.29 0.11 False
SerializeToUtf8Bytes - Duration of single invocation 1.44 ms 1.73 ms 1.20 0.06 False
SerializeObjectProperty - Duration of single invocation 1.47 ms 1.79 ms 1.21 0.09 False
SerializeToWriter - Duration of single invocation 1.41 ms 1.71 ms 1.21 0.08 False
SerializeToString - Duration of single invocation 928.36 μs 1.20 ms 1.30 0.12 False
SerializeToUtf8Bytes - Duration of single invocation 889.05 μs 1.15 ms 1.29 0.08 False
SerializeObjectProperty - Duration of single invocation 917.53 μs 1.20 ms 1.30 0.10 False
SerializeToString - Duration of single invocation 1.44 ms 1.82 ms 1.27 0.05 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1024863930555557 > 901.3729586266448.
IsChangePoint: Marked as a change because one of 3/10/2023 11:41:11 AM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.259682282447997 (T) = (0 -1123007.4771714562) / Math.Sqrt((418743005.78988874 / (43)) + (628950525.1830385 / (6))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (6) - 2, .025) and -0.3007620807913441 = (863345.7984016974 - 1123007.4771714562) / 863345.7984016974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7334409737103176 > 1.5106449220028406.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -19.30336757285662 (T) = (0 -1760532.8299248866) / Math.Sqrt((764249190.579451 / (43)) + (1911901496.1484249 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.23002579589065036 = (1431297.4864483236 - 1760532.8299248866) / 1431297.4864483236 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7883439810185184 > 1.554131670706731.
IsChangePoint: Marked as a change because one of 3/10/2023 4:14:40 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -16.830382284852913 (T) = (0 -1763616.5070400883) / Math.Sqrt((381766522.5330156 / (42)) + (2052467067.4214847 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.1989310502871279 = (1470990.7684997614 - 1763616.5070400883) / 1470990.7684997614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7108328703703704 > 1.4829773012784087.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -31.906590112677556 (T) = (0 -1684949.1084665533) / Math.Sqrt((1117112246.5810547 / (42)) + (359277974.52884793 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.20070805174327352 = (1403296.2517576392 - 1684949.1084665533) / 1403296.2517576392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2038464321428572 > 965.5757336914063.
IsChangePoint: Marked as a change because one of 2/13/2023 6:54:14 PM, 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -45.77574660563786 (T) = (0 -1182657.0629807692) / Math.Sqrt((439787795.285117 / (42)) + (154765105.23673412 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.2835820233641032 = (921372.4105306315 - 1182657.0629807692) / 921372.4105306315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.148270192628205 > 935.54041484375.
IsChangePoint: Marked as a change because one of 2/16/2023 3:28:45 AM, 3/10/2023 4:14:40 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -55.43342176315234 (T) = (0 -1144866.63070382) / Math.Sqrt((498266766.82401186 / (43)) + (62833427.953192644 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.281341170824307 = (893490.8647064772 - 1144866.63070382) / 893490.8647064772 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1962798625 > 960.3113865820311.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 3/10/2023 4:14:40 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -33.13704484439174 (T) = (0 -1187492.0079490452) / Math.Sqrt((307815688.6058221 / (43)) + (395338337.9171594 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.2863453064862063 = (923151.8177594243 - 1187492.0079490452) / 923151.8177594243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.822725446986607 > 1.548973780778846.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -15.867382020776489 (T) = (0 -1796059.0779696947) / Math.Sqrt((622096772.6900558 / (42)) + (2879577254.4005203 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.2230645613881314 = (1468490.817795617 - 1796059.0779696947) / 1468490.817795617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_ToString<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 1.46 ms 1.78 ms 1.22 0.07 False
SystemTextJson_SourceGen_ - Duration of single invocation 907.61 μs 1.15 ms 1.27 0.08 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<MyEventsListerViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7816976195238092 > 1.527581492286383.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -17.142555679411476 (T) = (0 -1775003.411711784) / Math.Sqrt((8725171752.854292 / (43)) + (688326514.5085528 / (6))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (6) - 2, .025) and -0.20790869212843227 = (1469484.7576467765 - 1775003.411711784) / 1469484.7576467765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

MicroBenchmarks.Serializers.Json_ToString<MyEventsListerViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1498641028846155 > 951.2720519140626.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -32.417678719110675 (T) = (0 -1169989.0603120094) / Math.Sqrt((373949062.6043899 / (43)) + (383176399.50074583 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.28317408347508105 = (911792.9323692815 - 1169989.0603120094) / 911792.9323692815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 27.52 μs 32.01 μs 1.16 0.04 False
SerializeToUtf8Bytes - Duration of single invocation 33.74 μs 35.95 μs 1.07 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<IndexViewModel>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.01199827380952 > 29.002805014581433.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 3/1/2023 3:09:21 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -21.30952675471099 (T) = (0 -32161.950693074992) / Math.Sqrt((123359.35798565281 / (43)) + (288230.3988306383 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.16151054533309847 = (27689.762113913115 - 32161.950693074992) / 27689.762113913115 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<IndexViewModel>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.9536878591954 > 33.46590119859738.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 3/1/2023 3:09:21 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -20.14335851154159 (T) = (0 -36358.61997125247) / Math.Sqrt((248191.0850875688 / (42)) + (301564.7202241952 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.13976317934000973 = (31900.153146117827 - 36358.61997125247) / 31900.153146117827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 564.19 ns 660.12 ns 1.17 0.18 False
SerializeToString - Duration of single invocation 943.74 ns 1.04 μs 1.10 0.25 False
SerializeToWriter - Duration of single invocation 596.49 ns 669.53 ns 1.12 0.20 False
SerializeObjectProperty - Duration of single invocation 1.45 μs 1.66 μs 1.14 0.25 False
SerializeToUtf8Bytes - Duration of single invocation 743.14 ns 889.25 ns 1.20 0.20 False
SerializeToString - Duration of single invocation 889.06 ns 1.05 μs 1.18 0.22 False
SerializeToUtf8Bytes - Duration of single invocation 773.45 ns 843.16 ns 1.09 0.23 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 660.1159389299919 > 596.0854349810462.
IsChangePoint: Marked as a change because one of 3/13/2023 8:52:40 AM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -17.52887284561596 (T) = (0 -665.5075915520725) / Math.Sqrt((349.4172701304029 / (43)) + (201.12870822786022 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.1903460961364246 = (559.0874735609661 - 665.5075915520725) / 559.0874735609661 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0409540723651711 > 960.3639619819999.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -14.965959027110165 (T) = (0 -1042.5649215164249) / Math.Sqrt((939.6907994466882 / (43)) + (306.034038676819 / (6))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (6) - 2, .025) and -0.13964026898846696 = (914.8193073607296 - 1042.5649215164249) / 914.8193073607296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 669.527917667613 > 596.9572439283145.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -18.771343205310938 (T) = (0 -667.3348530344057) / Math.Sqrt((254.69592280282856 / (43)) + (173.83269771302704 / (6))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (6) - 2, .025) and -0.19927545798075977 = (556.448352706232 - 667.3348530344057) / 556.448352706232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.657129068055832 > 1.5696686159442366.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -7.652353630095842 (T) = (0 -1688.1028963244996) / Math.Sqrt((1787.0238810917895 / (42)) + (3001.009135414357 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.10914851990090095 = (1521.980930448635 - 1688.1028963244996) / 1521.980930448635 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 889.248148732966 > 784.6956055090557.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -16.178954668021497 (T) = (0 -884.3877778185087) / Math.Sqrt((493.35994200494923 / (43)) + (364.40959673864944 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.17070740385640967 = (755.4302423519832 - 884.3877778185087) / 755.4302423519832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0456139683557115 > 933.6215003256035.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -10.959092994261272 (T) = (0 -1064.715372952346) / Math.Sqrt((802.894219685747 / (43)) + (1336.0516474424942 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.17508085541800214 = (906.0783928554458 - 1064.715372952346) / 906.0783928554458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 843.1623146075597 > 805.9544086248478.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -8.737477700212265 (T) = (0 -881.8324312037756) / Math.Sqrt((290.62529696233787 / (43)) + (1396.922050362772 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.16593766722221098 = (756.3289668003418 - 881.8324312037756) / 756.3289668003418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Globalization.Tests.Perf_DateTimeCultureInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 358.67 ns 580.46 ns 1.62 0.08 True
ToString - Duration of single invocation 356.11 ns 565.42 ns 1.59 0.10 True
ToString - Duration of single invocation 349.36 ns 558.50 ns 1.60 0.09 True
ToString - Duration of single invocation 344.73 ns 581.18 ns 1.69 0.12 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.Perf_DateTimeCultureInfo*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: fr)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 580.455666568876 > 364.66230665573227.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -58.53689009274283 (T) = (0 -574.4791172274151) / Math.Sqrt((160.86267299196928 / (43)) + (74.31578336912965 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6289191957227552 = (352.67502447996964 - 574.4791172274151) / 352.67502447996964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: )

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 565.4241460552668 > 369.19276072376414.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -33.33524350429814 (T) = (0 -587.8731031250215) / Math.Sqrt((39.81464781494948 / (42)) + (349.33880517638755 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.6788987793365895 = (350.15398805478753 - 587.8731031250215) / 350.15398805478753 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: ja)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 558.4961977774478 > 367.85619242284764.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -73.64962535247909 (T) = (0 -570.4353045781777) / Math.Sqrt((48.80038574832103 / (43)) + (52.90397976873987 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6146262492733631 = (353.292475478392 - 570.4353045781777) / 353.292475478392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: da)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 581.1806198994343 > 371.1137097895988.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -19.421235468280077 (T) = (0 -590.6690485025102) / Math.Sqrt((90.01340326392385 / (43)) + (1043.273125159857 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6784550424032257 = (351.9123441380863 - 590.6690485025102) / 351.9123441380863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 1.38 ms 1.72 ms 1.25 0.06 False
SystemTextJson_SourceGen_ - Duration of single invocation 1.30 ms 1.60 ms 1.23 0.07 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7246685162037043 > 1.4672714498997175.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -22.448551132325488 (T) = (0 -1691423.309335212) / Math.Sqrt((531703895.71895164 / (43)) + (1054205304.6076332 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.20398913058443383 = (1404849.3182942362 - 1691423.309335212) / 1404849.3182942362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6022623655913977 > 1.3721657819213418.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -16.95924254476948 (T) = (0 -1600796.5826501288) / Math.Sqrt((1284416405.5164037 / (41)) + (1656159385.6116073 / (7))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (7) - 2, .025) and -0.20978955255254847 = (1323202.5183823006 - 1600796.5826501288) / 1323202.5183823006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Append_ValueTypes - Duration of single invocation 6.30 μs 6.82 μs 1.08 0.10 False
Append_ValueTypes_Interpolated - Duration of single invocation 8.45 μs 9.78 μs 1.16 0.12 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.Append_ValueTypes

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.817022412823868 > 6.409512357608278.
IsChangePoint: Marked as a change because one of 2/12/2023 3:00:19 AM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -10.694344103194501 (T) = (0 -6969.682413049682) / Math.Sqrt((19648.708733229687 / (43)) + (37512.4582635502 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.1325244710812947 = (6154.111978167918 - 6969.682413049682) / 6154.111978167918 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Tests.Perf_StringBuilder.Append_ValueTypes_Interpolated

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.778222395506385 > 8.870198687098206.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -38.543168387420515 (T) = (0 -9749.466830629566) / Math.Sqrt((28780.18582165279 / (43)) + (3108.6513994967463 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.15196008518214554 = (8463.372087313252 - 9749.466830629566) / 8463.372087313252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_DateTimes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDateTimes - Duration of single invocation 7.27 ms 15.21 ms 2.09 0.11 True
WriteDateTimes - Duration of single invocation 8.37 ms 15.63 ms 1.87 0.06 True
WriteDateTimes - Duration of single invocation 7.36 ms 15.01 ms 2.04 0.11 True
WriteDateTimes - Duration of single invocation 8.46 ms 15.49 ms 1.83 0.09 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_DateTimes*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.213145820833333 > 7.70143438125.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -34.920054594525325 (T) = (0 -15340341.68857448) / Math.Sqrt((37287439335.00616 / (42)) + (371548600378.466 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -1.1222916700892094 = (7228196.719977541 - 15340341.68857448) / 7228196.719977541 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: True, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.629066666666668 > 8.826791964843752.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -25.617238810746446 (T) = (0 -16247207.360770974) / Math.Sqrt((14463057281.009233 / (43)) + (652324797797.6385 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.9312130303363065 = (8412954.503492368 - 16247207.360770974) / 8412954.503492368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.006017768888888 > 7.5158398423076935.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -66.5256048402972 (T) = (0 -15111659.094716337) / Math.Sqrt((61407129919.42048 / (42)) + (91044348727.34425 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -1.1255148638444077 = (7109646.397571625 - 15111659.094716337) / 7109646.397571625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: True, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.494484431111115 > 8.8966080478125.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -18.580510834604535 (T) = (0 -16015703.269991279) / Math.Sqrt((101391066844.01012 / (42)) + (1096545253234.7323 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.8611475292152594 = (8605284.115625264 - 16015703.269991279) / 8605284.115625264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterUInt64 - Duration of single invocation 32.76 ns 147.93 ns 4.52 0.45 False
FormatterInt64 - Duration of single invocation 36.90 ns 136.44 ns 3.70 0.40 False
FormatterInt64 - Duration of single invocation 33.22 ns 140.47 ns 4.23 0.42 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 18446744073709551615)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.9263102204153 > 35.412342602779404.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -94.97110814822713 (T) = (0 -147.71731017211124) / Math.Sqrt((6.301747962894654 / (42)) + (8.780595259255916 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -3.2002088922817444 = (35.16903895983718 - 147.71731017211124) / 35.16903895983718 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 136.44015653580198 > 35.792598546829836.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -175.60890137433225 (T) = (0 -138.32274463793962) / Math.Sqrt((3.246279537751629 / (43)) + (1.924934879637 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -3.0257993343610163 = (34.35907583801479 - 138.32274463793962) / 34.35907583801479 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 140.4692937668984 > 34.36084923962718.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -136.3942956826316 (T) = (0 -140.87140646759366) / Math.Sqrt((5.94395574219608 / (43)) + (3.3382232232870894 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -3.155759103320809 = (33.89787592717424 - 140.87140646759366) / 33.89787592717424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 1411364699b5784040e86f76cb3db8200f6a2c8c
Compare 86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Tests.Perf_DateTime

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 360.58 ns 599.99 ns 1.66 0.16 True
ToString - Duration of single invocation 130.53 ns 185.18 ns 1.42 0.25 False
ToString - Duration of single invocation 358.66 ns 569.28 ns 1.59 0.15 True
ToString - Duration of single invocation 370.78 ns 610.66 ns 1.65 0.13 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_DateTime*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_DateTime.ToString(format: null)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 599.9888331743136 > 377.39721273111485.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.322655055724343 (T) = (0 -601.8257350283385) / Math.Sqrt((151.86787165916476 / (43)) + (668.1806228612865 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6725035176669795 = (359.8352581451318 - 601.8257350283385) / 359.8352581451318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: "o")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 185.17668575963293 > 136.5719478509588.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -26.270893599250268 (T) = (0 -178.7206589105019) / Math.Sqrt((26.735841185961704 / (43)) + (21.936082818519164 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.3983300571957398 = (127.81006743780834 - 178.7206589105019) / 127.81006743780834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: "G")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 569.2818855012036 > 363.7887397283544.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.932149334810706 (T) = (0 -601.256363345744) / Math.Sqrt((123.23113972028878 / (43)) + (659.2188534817575 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6905721339501865 = (355.65259314954517 - 601.256363345744) / 355.65259314954517 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: "s")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 610.6603128857563 > 389.66698580220293.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -18.132431961749482 (T) = (0 -561.851156518624) / Math.Sqrt((150.855263664341 / (42)) + (702.6031308587686 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.49044704835091574 = (376.96821040390284 - 561.851156518624) / 376.96821040390284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@lewing
Copy link
Member

lewing commented Apr 11, 2023 •
edited
Loading

dotnet/runtime#84469 but for aot this time

cc @stephentoub

@stephentoub
Copy link
Member

Same as #15660 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stephentoub stephentoub

Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression untriaged wasm-aot
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lewing @stephentoub