Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 87 Regressions on 8/17/2022 6:09:21 PM #7976

Closed
performanceautofiler bot opened this issue Aug 23, 2022 · 14 comments
Closed

[Perf] Linux/x64: 87 Regressions on 8/17/2022 6:09:21 PM #7976

performanceautofiler bot opened this issue Aug 23, 2022 · 14 comments
Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval.

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Aug 23, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfValue - Duration of single invocation 2.00 μs 4.42 μs 2.22 0.05 False
IndexOfAnyFiveValues - Duration of single invocation 5.98 μs 16.58 μs 2.77 0.03 False
LastIndexOfValue - Duration of single invocation 1.76 μs 4.38 μs 2.50 0.08 False
IndexOfAnyThreeValues - Duration of single invocation 4.72 μs 9.95 μs 2.11 0.06 False
IndexOfAnyFourValues - Duration of single invocation 5.15 μs 16.80 μs 3.26 0.02 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfValue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.424370395797761 > 2.099354007601711.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -228.9822018722831 (T) = (0 -4440.129849755984) / Math.Sqrt((746.1326660454284 / (16)) + (1921.3579389411896 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.2120651354916854 = (2007.2328696456113 - 4440.129849755984) / 2007.2328696456113 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyFiveValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -230.9342858971992 (T) = (0 -16559.50904449143) / Math.Sqrt((83.31507440812801 / (2)) + (51518.820613645454 / (25))) is less than -2.0595385527466985 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (2) + (25) - 2, .025) and -1.7734624046482563 = (5970.6989417769255 - 16559.50904449143) / 5970.6989417769255 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.382803354060675 > 1.8459576629846874.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -608.8437463151625 (T) = (0 -4393.6243695298745) / Math.Sqrt((83.06115909458961 / (16)) + (378.2744720148661 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.4953964456371467 = (1760.6919242076806 - 4393.6243695298745) / 1760.6919242076806 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.953077892514834 > 4.958038374220887.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -133.06334302856706 (T) = (0 -10035.178960318004) / Math.Sqrt((10566.657385247858 / (16)) + (27466.34550316625 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.1350840643169582 = (4700.132949345201 - 10035.178960318004) / 4700.132949345201 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyFourValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.79938062185671 > 5.4029949952717935.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -120.59949876258965 (T) = (0 -16744.636886163873) / Math.Sqrt((144.12597451887595 / (16)) + (249584.24096087858 / (27))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (27) - 2, .025) and -2.255209824331321 = (5143.9500953225115 - 16744.636886163873) / 5143.9500953225115 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 3.91 μs 8.65 μs 2.21 0.06 False
IndexOfString - Duration of single invocation 578.96 ns 1.03 μs 1.79 0.04 False
IndexOfString - Duration of single invocation 1.04 μs 2.35 μs 2.26 0.06 False
IndexOfString - Duration of single invocation 511.54 ns 677.87 ns 1.33 0.11 False
IndexOfString - Duration of single invocation 3.90 μs 8.63 μs 2.21 0.05 False
IndexOfString - Duration of single invocation 1.60 μs 3.37 μs 2.10 0.06 False
IndexOfString - Duration of single invocation 388.82 ns 415.19 ns 1.07 0.12 False
IndexOfString - Duration of single invocation 587.29 ns 1.07 μs 1.82 0.07 False
IndexOfString - Duration of single invocation 366.00 ns 495.09 ns 1.35 0.13 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.64741195028587 > 4.1013109440625.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -180.96434391360944 (T) = (0 -8706.652358439575) / Math.Sqrt((1324.2543448862339 / (16)) + (17940.025527011872 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.2244650627025373 = (3914.0432027562288 - 8706.652358439575) / 3914.0432027562288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbcbbbbbbbbbbbbbbbbbbba!", value: "y", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0348012839410983 > 603.3218214752392.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -214.46461206020024 (T) = (0 -1023.6016826715849) / Math.Sqrt((36.39556865114193 / (16)) + (62.77360309679295 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.7903560279728833 = (571.7307991699003 - 1023.6016826715849) / 571.7307991699003 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.351876672460109 > 1.1005424139574933.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -138.67827147386146 (T) = (0 -2357.7106617874088) / Math.Sqrt((525.5180591270678 / (16)) + (1612.733691558525 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.2381400077496516 = (1053.424117179327 - 2357.7106617874088) / 1053.424117179327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "More Test's", value: "Tests", comparisonType: OrdinalIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 677.8736925409745 > 536.477630083975.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -38.38513981916017 (T) = (0 -664.3106228947553) / Math.Sqrt((198.4434352488016 / (16)) + (72.66655300880606 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.2882830899330542 = (515.6557810048382 - 664.3106228947553) / 515.6557810048382 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.630530967689898 > 4.096269678979396.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -260.7095987365435 (T) = (0 -8688.069538319753) / Math.Sqrt((1323.1846778303511 / (16)) + (7342.706292114539 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.2220554931170406 = (3909.9246464508224 - 8688.069538319753) / 3909.9246464508224 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3717654301364943 > 1.6824693754498397.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -132.24447565482643 (T) = (0 -3398.9921041228167) / Math.Sqrt((2351.302386780355 / (16)) + (982.3132722364186 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.1049970280982135 = (1614.7253695620082 - 3398.9921041228167) / 1614.7253695620082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 415.18956022735387 > 410.1973734939151.
IsChangePoint: Marked as a change because one of 6/17/2022 8:24:28 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -4.450843903159241 (T) = (0 -427.223316492107) / Math.Sqrt((104.43290618479654 / (16)) + (1179.447940174716 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.07835981751772005 = (396.17881671030153 - 427.223316492107) / 396.17881671030153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだすだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "す", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.069781383349324 > 614.5315261211443.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -147.23790642148083 (T) = (0 -1067.3376950804065) / Math.Sqrt((127.05744436804608 / (16)) + (81.3158167840278 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.8254564622692572 = (584.696330557005 - 1067.3376950804065) / 584.696330557005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 495.0899798707861 > 384.86005499687917.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -27.43734624155144 (T) = (0 -498.9181598445892) / Math.Sqrt((97.6860874663925 / (16)) + (523.0212412338225 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.3702401017648088 = (364.1100265581226 - 498.9181598445892) / 364.1100265581226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 1.35 ms 1.62 ms 1.20 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.619453370673077 > 1.4160329543694912.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -52.032986115816996 (T) = (0 -1586243.5162880719) / Math.Sqrt((152836733.53902644 / (16)) + (317923584.2416149 / (27))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (27) - 2, .025) and -0.17853275682885175 = (1345947.753337185 - 1586243.5162880719) / 1345947.753337185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfChar - Duration of single invocation 643.14 ns 1.28 μs 1.99 0.06 False
IndexOfShort - Duration of single invocation 644.02 ns 1.27 μs 1.97 0.01 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.IndexOfChar

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2804434792942432 > 674.4623342442469.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -275.8648770624791 (T) = (0 -1270.0653190761518) / Math.Sqrt((27.01499653376699 / (16)) + (104.50117188523133 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.9986530990935593 = (635.4606107744056 - 1270.0653190761518) / 635.4606107744056 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.IndexOfShort

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2715719728990553 > 673.0728550867216.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -210.83791530025124 (T) = (0 -1271.2441752009593) / Math.Sqrt((30.694512461879636 / (16)) + (206.4508123471615 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.9943664505987144 = (637.4175492268877 - 1271.2441752009593) / 637.4175492268877 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Aug 23, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Utf8JsonReaderCommentsTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Utf8JsonReaderCommentParsing - Duration of single invocation 313.54 μs 417.08 μs 1.33 0.03 False
Utf8JsonReaderCommentParsing - Duration of single invocation 427.46 μs 768.80 μs 1.80 0.02 False
Utf8JsonReaderCommentParsing - Duration of single invocation 1.71 μs 1.92 μs 1.12 0.08 False
Utf8JsonReaderCommentParsing - Duration of single invocation 42.87 μs 94.52 μs 2.20 0.02 False
Utf8JsonReaderCommentParsing - Duration of single invocation 22.42 μs 73.08 μs 3.26 0.04 False
Utf8JsonReaderCommentParsing - Duration of single invocation 45.11 μs 96.79 μs 2.15 0.02 False
Utf8JsonReaderCommentParsing - Duration of single invocation 22.27 μs 74.59 μs 3.35 0.02 False
Utf8JsonReaderCommentParsing - Duration of single invocation 1.78 μs 2.02 μs 1.14 0.05 False
Utf8JsonReaderCommentParsing - Duration of single invocation 2.15 μs 2.39 μs 1.11 0.06 False
Utf8JsonReaderCommentParsing - Duration of single invocation 434.94 μs 781.26 μs 1.80 0.03 False
Utf8JsonReaderCommentParsing - Duration of single invocation 313.81 μs 416.77 μs 1.33 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongMultiLine)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 417.076616554054 > 329.5719264283655.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -102.57564960113481 (T) = (0 -418632.32594543713) / Math.Sqrt((1928136.749094154 / (16)) + (25624554.91591094 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.3321921240860597 = (314243.20739971095 - 418632.32594543713) / 314243.20739971095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: LongMultiLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 768.7963426870749 > 450.35074124599356.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -190.18394581898855 (T) = (0 -777104.2962117526) / Math.Sqrt((28734077.789752726 / (16)) + (45828526.847630955 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.8171361206846663 = (427653.3207203832 - 777104.2962117526) / 427653.3207203832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: ShortSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.919150241328703 > 1.7754853388550547.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -15.250102491803851 (T) = (0 -1911.705582521309) / Math.Sqrt((2245.943659709621 / (16)) + (1248.3270590086538 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.1216907120601176 = (1704.307222986839 - 1911.705582521309) / 1704.307222986839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 94.51739551282051 > 45.39014446090988.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -246.95094150628128 (T) = (0 -95377.5486184878) / Math.Sqrt((354567.6382718937 / (16)) + (642319.9443519037 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.1979001806389102 = (43394.849983934386 - 95377.5486184878) / 43394.849983934386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.07638398634076 > 23.543713681752877.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -249.47830350147433 (T) = (0 -73669.95007150831) / Math.Sqrt((7560.770973549991 / (16)) + (1208713.5759309873 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -2.2815882258212827 = (22449.480252224803 - 73669.95007150831) / 22449.480252224803 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: LongSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 96.79029472049692 > 47.48455000751264.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -365.2823731831631 (T) = (0 -97274.20805196515) / Math.Sqrt((160675.14757247726 / (15)) + (273139.17868400324 / (29))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (29) - 2, .025) and -1.140363656322024 = (45447.51438132715 - 97274.20805196515) / 45447.51438132715 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.59279740079367 > 23.407842403794646.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -410.30448639350595 (T) = (0 -73355.04280275121) / Math.Sqrt((1228.363536409068 / (16)) + (431389.0459670381 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -2.289507710460597 = (22299.702344360834 - 73355.04280275121) / 22299.702344360834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: ShortSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0206502732240437 > 1.8787082639433914.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -19.345225481857835 (T) = (0 -2037.8548943951093) / Math.Sqrt((1008.714254459033 / (16)) + (2338.9694853502683 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.12840837565993174 = (1805.9551296783864 - 2037.8548943951093) / 1805.9551296783864 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: ShortSingleLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.385919717501148 > 2.292540428646238.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -18.17365698748974 (T) = (0 -2437.1498976914727) / Math.Sqrt((1397.0710336766404 / (16)) + (3224.613015996631 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.11739770083167599 = (2181.0944267000987 - 2437.1498976914727) / 2181.0944267000987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongMultiLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 781.2607142857144 > 442.33662132601347.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -194.57767438088297 (T) = (0 -771828.3875788273) / Math.Sqrt((22186903.18980836 / (16)) + (53780207.542337514 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.8310370607174161 = (421525.2679137026 - 771828.3875788273) / 421525.2679137026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongMultiLine)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 416.76790529279276 > 329.61014969107146.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -53.73178844669315 (T) = (0 -418864.1733631213) / Math.Sqrt((44359038.289509095 / (16)) + (23562133.366976757 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.32251452630399274 = (316718.01332399226 - 418864.1733631213) / 316718.01332399226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Replace_String - Duration of single invocation 2.39 μs 2.98 μs 1.25 0.01 False
Replace_String - Duration of single invocation 1.73 μs 1.92 μs 1.11 0.01 False
Replace_Char - Duration of single invocation 150.92 ns 180.99 ns 1.20 0.15 False
Replace_Char - Duration of single invocation 2.30 μs 2.47 μs 1.07 0.04 False
Replace_Char - Duration of single invocation 318.66 ns 572.63 ns 1.80 0.08 False
IndexOfAny - Duration of single invocation 895.08 ns 2.11 μs 2.36 0.04 False
Format_MultipleArgs - Duration of single invocation 7.80 μs 8.77 μs 1.13 0.02 False
Replace_String - Duration of single invocation 464.19 ns 698.35 ns 1.50 0.08 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9834135878728927 > 2.5031605977991176.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -62.81876740473499 (T) = (0 -2985.045253916247) / Math.Sqrt((1018.4841881911484 / (16)) + (823.2163408555915 / (24))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (24) - 2, .025) and -0.2630776361475241 = (2363.3109861883436 - 2985.045253916247) / 2363.3109861883436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.916961297433969 > 1.7886904642661676.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -26.74311478194232 (T) = (0 -1921.865172165745) / Math.Sqrt((377.97604225187206 / (16)) + (1227.1010184100255 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.12738788155890926 = (1704.7062538123637 - 1921.865172165745) / 1704.7062538123637 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'a', newChar: 'b')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.99191089085383 > 152.44948038655173.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -25.683760352548866 (T) = (0 -181.05607911516873) / Math.Sqrt((35.48082109890826 / (16)) + (4.534171317426874 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.2796939007713365 = (141.48389627084808 - 181.05607911516873) / 141.48389627084808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.468843295759652 > 2.4180251807238244.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -15.99026679562764 (T) = (0 -2477.4025830727223) / Math.Sqrt((781.5059316014139 / (16)) + (1701.124826717512 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.07172206673955386 = (2311.609194172515 - 2477.4025830727223) / 2311.609194172515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 572.6288268598523 > 332.30943791185496.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -129.8317847496882 (T) = (0 -574.1748989444015) / Math.Sqrt((50.94620966571908 / (16)) + (23.369555858879608 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.8237234534597707 = (314.8366041217154 - 574.1748989444015) / 314.8366041217154 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.IndexOfAny

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1107680258711237 > 941.2799038134441.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -217.55344561937406 (T) = (0 -2114.6424450181576) / Math.Sqrt((96.26321391000975 / (16)) + (701.131993933497 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.343755332566345 = (902.245390393495 - 2114.6424450181576) / 902.245390393495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Format_MultipleArgs

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.772714981070846 > 8.164352893033962.
IsChangePoint: Marked as a change because one of 6/6/2022 10:51:35 AM, 7/2/2022 7:35:06 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -19.906394423534817 (T) = (0 -8410.834220877343) / Math.Sqrt((9836.509564061798 / (16)) + (13077.40903064493 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.08441697487861992 = (7756.088677806595 - 8410.834220877343) / 7756.088677806595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 698.347144796902 > 484.3843890836618.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -98.89436491006387 (T) = (0 -705.7927231376733) / Math.Sqrt((76.69953413638248 / (16)) + (43.54270970215507 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.5421616896255232 = (457.66454184778644 - 705.7927231376733) / 457.66454184778644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLine - Duration of single invocation 4.35 ms 4.61 ms 1.06 0.11 False
ReadLine - Duration of single invocation 221.78 μs 524.84 μs 2.37 0.05 False
ReadLine - Duration of single invocation 606.35 μs 921.13 μs 1.52 0.04 False
ReadLine - Duration of single invocation 3.88 ms 4.11 ms 1.06 0.07 False
ReadLine - Duration of single invocation 194.93 μs 512.00 μs 2.63 0.06 False
ReadLine - Duration of single invocation 220.04 μs 533.69 μs 2.43 0.07 False
ReadLine - Duration of single invocation 313.73 μs 616.24 μs 1.96 0.05 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StringReaderReadLineTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.61009134375 > 4.576353383037861.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -5.173919239348665 (T) = (0 -4691738.895104286) / Math.Sqrt((28417539521.29462 / (16)) + (19005525056.775864 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.05750445690741219 = (4436613.826503298 - 4691738.895104286) / 4436613.826503298 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 524.8363098214285 > 232.70500897294826.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -62.694577839828376 (T) = (0 -528617.5913092428) / Math.Sqrt((300674386.1890367 / (15)) + (89228276.20296077 / (28))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (28) - 2, .025) and -1.3345287639020793 = (226434.38773728273 - 528617.5913092428) / 226434.38773728273 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 921.1302499999998 > 646.1456844230769.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -56.12836860970813 (T) = (0 -925387.7938526968) / Math.Sqrt((184497933.06426266 / (16)) + (547218785.6514761 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.5024636727936888 = (615913.5895326014 - 925387.7938526968) / 615913.5895326014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [   1,    1])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.114136458333333 > 4.079841063007812.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -7.80557344135814 (T) = (0 -4174660.182681525) / Math.Sqrt((18094851290.875774 / (16)) + (6152484362.115495 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.07356357968910857 = (3888600.7886840366 - 4174660.182681525) / 3888600.7886840366 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 511.9964336228288 > 204.62784696952164.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -375.11959356902184 (T) = (0 -513160.2349164922) / Math.Sqrt((6430215.36917245 / (16)) + (8776794.116398722 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.6196181701645114 = (195891.23360076017 - 513160.2349164922) / 195891.23360076017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 533.694711698718 > 231.1549263949079.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -62.55159999578835 (T) = (0 -537327.1636288177) / Math.Sqrt((7474496.969665847 / (16)) + (626751355.9623219 / (25))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (25) - 2, .025) and -1.4288532662318103 = (221226.68795980493 - 537327.1636288177) / 221226.68795980493 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 616.236696474359 > 328.7316166796876.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -152.65719828076752 (T) = (0 -621195.5382827738) / Math.Sqrt((17403458.669043742 / (16)) + (79805212.6819289 / (27))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (27) - 2, .025) and -0.9769017080538159 = (314226.82056070305 - 621195.5382827738) / 314226.82056070305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Aug 23, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 2.27 ms 2.50 ms 1.10 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.500688245535715 > 2.409311259117445.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -10.875523016320345 (T) = (0 -2507088.2371718856) / Math.Sqrt((5132018085.1523285 / (16)) + (301732743.1397107 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.08575769440484561 = (2309067.8980139648 - 2507088.2371718856) / 2309067.8980139648 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 49.07 μs 54.43 μs 1.11 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<SimpleStructWithProperties>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.43084206002332 > 51.51931005306603.
IsChangePoint: Marked as a change because one of 6/17/2022 12:17:51 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -28.561151966906632 (T) = (0 -53859.64023973227) / Math.Sqrt((355044.4473096438 / (16)) + (83546.48345443682 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.0918668359909751 = (49328.03018130816 - 53859.64023973227) / 49328.03018130816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Net.Tests.Perf_WebUtility

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HtmlDecode_Entities - Duration of single invocation 7.36 μs 8.32 μs 1.13 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Tests.Perf_WebUtility*'

Payloads

Baseline
Compare

Histogram

System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.317603445213996 > 7.56699718392585.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -25.91434084455259 (T) = (0 -8218.05309067935) / Math.Sqrt((7430.371084409863 / (15)) + (31151.15102817964 / (29))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (29) - 2, .025) and -0.14276199601001058 = (7191.395163098651 - 8218.05309067935) / 7191.395163098651 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Delete - Duration of single invocation 13.58 μs 14.81 μs 1.09 0.19 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_File*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_File.Delete

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.805555555555555 > 13.934117647058825.
IsChangePoint: Marked as a change because one of 6/18/2022 12:36:47 AM, 7/26/2022 1:50:03 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -11.471955243799224 (T) = (0 -14665.776425512737) / Math.Sqrt((150002.53146019988 / (16)) + (92733.57848135126 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.09614278259687342 = (13379.43984885621 - 14665.776425512737) / 13379.43984885621 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 10.40 ms 12.57 ms 1.21 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<CollectionsOfPrimitives>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.566868409774438 > 10.763696082391306.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -46.919617963579846 (T) = (0 -12653622.568551008) / Math.Sqrt((24281814299.226254 / (16)) + (30639266325.159958 / (27))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (27) - 2, .025) and -0.23604410122093109 = (10237193.44322108 - 12653622.568551008) / 10237193.44322108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Logging.FormattingOverhead

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TwoArguments_DefineMessage - Duration of single invocation 3.92 μs 4.23 μs 1.08 0.06 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Logging.FormattingOverhead*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Logging.FormattingOverhead.TwoArguments_DefineMessage

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.228927996743399 > 4.11781354055331.
IsChangePoint: Marked as a change because one of 6/6/2022 10:51:35 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -10.028250355461715 (T) = (0 -4180.690415520118) / Math.Sqrt((5582.098372535155 / (16)) + (5378.797626221735 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.058703962900710095 = (3948.8757594385256 - 4180.690415520118) / 3948.8757594385256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Aug 23, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<ClassImplementingIXmlSerialiable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 27.14 μs 29.74 μs 1.10 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;ClassImplementingIXmlSerialiable&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<ClassImplementingIXmlSerialiable>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.7449048015873 > 28.475700524983353.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -36.26240557061637 (T) = (0 -29501.43718058425) / Math.Sqrt((36731.27730432557 / (16)) + (56741.85655938553 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.08791452357577142 = (27117.421949307696 - 29501.43718058425) / 27117.421949307696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 22.29 μs 24.24 μs 1.09 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<XmlElement>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.242818343777728 > 23.420282569972155.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -31.135417474086633 (T) = (0 -24363.76215450808) / Math.Sqrt((48676.718132044334 / (16)) + (54545.60703830902 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.09849923684115346 = (22179.134347483538 - 24363.76215450808) / 22179.134347483538 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOf - Duration of single invocation 254.39 ns 303.75 ns 1.19 0.19 False
LastIndexOf - Duration of single invocation 193.75 ns 222.13 ns 1.15 0.07 False
IndexOfAny - Duration of single invocation 274.40 ns 312.02 ns 1.14 0.15 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.IndexOf

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.75312907391657 > 259.9725405611434.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -35.87137659935671 (T) = (0 -308.6282272335256) / Math.Sqrt((16.432279761390557 / (16)) + (61.55651569538155 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.25988021237452424 = (244.966326323871 - 308.6282272335256) / 244.966326323871 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 222.13241024238366 > 202.8217296235999.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -19.968138571243166 (T) = (0 -226.09575694379947) / Math.Sqrt((4.51084292561241 / (16)) + (65.20789729306115 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.16645569228470333 = (193.8314146343203 - 226.09575694379947) / 193.8314146343203 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Microsoft.Extensions.Primitives.StringSegmentBenchmark.IndexOfAny

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 312.01574533057817 > 291.4121051847357.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -15.861317146866144 (T) = (0 -319.2223726798385) / Math.Sqrt((41.5586552553094 / (16)) + (114.66812625560945 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.1457104960005385 = (278.6239401613097 - 319.2223726798385) / 278.6239401613097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SetEnvironmentVariable - Duration of single invocation 1.90 μs 2.62 μs 1.38 0.03 False
GetEnvironmentVariable - Duration of single invocation 1.26 μs 1.56 μs 1.24 0.04 False
ExpandEnvironmentVariables - Duration of single invocation 3.44 μs 4.02 μs 1.17 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.SetEnvironmentVariable

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6163876759477342 > 1.984660467392358.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -98.2437070794776 (T) = (0 -2621.083301913633) / Math.Sqrt((535.1928917092663 / (16)) + (672.0546770424587 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.3928515437647343 = (1881.8109608645834 - 2621.083301913633) / 1881.8109608645834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.GetEnvironmentVariable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5619126906064986 > 1.2999614170472453.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -52.376588273066076 (T) = (0 -1556.0396899626862) / Math.Sqrt((527.5555108381092 / (16)) + (265.8561224748713 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.2795993202018014 = (1216.0366650689439 - 1556.0396899626862) / 1216.0366650689439 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.ExpandEnvironmentVariables

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.024084259858486 > 3.522412314446341.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -26.377989378054707 (T) = (0 -4008.261090778978) / Math.Sqrt((7692.257648599997 / (16)) + (4559.181990003355 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.1993602791710388 = (3341.9991977301147 - 4008.261090778978) / 3341.9991977301147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 77.97 μs 83.11 μs 1.07 0.00 False
DataContractSerializer_ - Duration of single invocation 49.58 μs 61.92 μs 1.25 0.01 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.1079548573975 > 81.9259716530338.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -31.372250094401643 (T) = (0 -83485.83388254583) / Math.Sqrt((147943.43849549265 / (16)) + (570458.8011175494 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.06826396585936338 = (78150.94073250498 - 83485.83388254583) / 78150.94073250498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;.DataContractSerializer_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.922778450963726 > 52.27067584440856.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -88.66592867829671 (T) = (0 -62311.483192904896) / Math.Sqrt((114669.3008278356 / (16)) + (344499.18431104993 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.24436545312109767 = (50074.90607893061 - 62311.483192904896) / 50074.90607893061 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 56.34 μs 62.20 μs 1.10 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.204662082439484 > 59.776548988553415.
IsChangePoint: Marked as a change because one of 6/17/2022 8:24:28 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -34.0269744674233 (T) = (0 -62451.47672903438) / Math.Sqrt((280910.29646721267 / (16)) + (241282.0894790869 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.09606704141182142 = (56977.78910365913 - 62451.47672903438) / 56977.78910365913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOf_Word_NotFound - Duration of single invocation 3.04 μs 7.12 μs 2.34 0.04 False
LastIndexOf_Word_NotFound - Duration of single invocation 2.13 μs 4.36 μs 2.04 0.04 False
IndexOf_Word_NotFound - Duration of single invocation 2.30 μs 4.38 μs 1.91 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.118387508761171 > 3.182182864156933.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -197.22136931401988 (T) = (0 -7145.780106725725) / Math.Sqrt((219.71236339860948 / (16)) + (12253.483118668653 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.3611319313419288 = (3026.421358278139 - 7145.780106725725) / 3026.421358278139 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.358033302979359 > 2.202654212599228.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -281.9936122004356 (T) = (0 -4375.812085460545) / Math.Sqrt((411.3918712032901 / (16)) + (1116.516821014151 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.0916344826922386 = (2092.0539040971616 - 4375.812085460545) / 2092.0539040971616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.384989886822578 > 2.4117003099811902.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -210.3105675362864 (T) = (0 -4402.356972613) / Math.Sqrt((249.5553694766067 / (16)) + (2490.099550850191 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.9275392828994649 = (2283.9259420907038 - 4402.356972613) / 2283.9259420907038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 18.83 ms 20.99 ms 1.11 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.985988894444446 > 20.088463316715977.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -32.79236237719975 (T) = (0 -21009217.220016956) / Math.Sqrt((38197156299.83066 / (16)) + (39733586351.16035 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.10580003268469074 = (18999110.688222915 - 21009217.220016956) / 18999110.688222915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.Perf_StreamWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteFormat - Duration of single invocation 2.55 μs 3.00 μs 1.18 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_StreamWriter*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_StreamWriter.WriteFormat

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0029341009621593 > 2.7154029798724673.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -42.10907206010271 (T) = (0 -3016.836117126905) / Math.Sqrt((610.0345776879633 / (16)) + (1941.2651929667438 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.16696060803693374 = (2585.2081864201396 - 3016.836117126905) / 2585.2081864201396 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 11.80 ms 14.98 ms 1.27 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.9826813627451 > 12.59120625230263.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -45.81703926070371 (T) = (0 -15176174.963003974) / Math.Sqrt((42942003566.13368 / (16)) + (52375546915.0276 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.25587521768837923 = (12084142.3966769 - 15176174.963003974) / 12084142.3966769 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 24.02 ms 27.01 ms 1.12 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.008111118518524 > 25.269387486.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -36.704062211535536 (T) = (0 -26945026.17702974) / Math.Sqrt((43515640128.383255 / (16)) + (79329784996.19202 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.11186505150152153 = (24234079.613026552 - 26945026.17702974) / 24234079.613026552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Aug 23, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReplaceWords - Duration of single invocation 243.66 μs 382.74 μs 1.57 0.00 False
MatchesWord - Duration of single invocation 115.32 μs 184.88 μs 1.60 0.02 False
SplitWords - Duration of single invocation 230.46 μs 371.49 μs 1.61 0.02 False
MatchWord - Duration of single invocation 7.84 μs 9.88 μs 1.26 0.01 False
MatchesWord - Duration of single invocation 78.69 μs 107.07 μs 1.36 0.04 False
ReplaceWords - Duration of single invocation 195.49 μs 255.66 μs 1.31 0.02 False
SplitWords - Duration of single invocation 177.95 μs 246.98 μs 1.39 0.01 False
MatchesWord - Duration of single invocation 75.85 μs 106.35 μs 1.40 0.02 False
MatchesWords - Duration of single invocation 225.32 μs 375.12 μs 1.66 0.01 False
MatchesWords - Duration of single invocation 173.60 μs 239.45 μs 1.38 0.01 False
ReplaceWords - Duration of single invocation 1.50 ms 1.64 ms 1.09 0.03 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 382.74268313008133 > 256.93320505371094.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -133.93826111582817 (T) = (0 -384743.8890934095) / Math.Sqrt((3356502.534623927 / (15)) + (24990473.569605228 / (29))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (29) - 2, .025) and -0.5691233120260384 = (245196.71981460243 - 384743.8890934095) / 245196.71981460243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 184.8819165730337 > 118.700188493143.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -91.33738775491729 (T) = (0 -177361.7613408131) / Math.Sqrt((1806984.6492833847 / (16)) + (10544684.81276144 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.5632633062719897 = (113456.10213533293 - 177361.7613408131) / 113456.10213533293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 371.4890519770408 > 248.71524123106065.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -126.02562638330163 (T) = (0 -375845.53227303847) / Math.Sqrt((8579529.811801046 / (16)) + (19795021.757936306 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.597105511214483 = (235329.181218112 - 375845.53227303847) / 235329.181218112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.87709388994508 > 8.416294006205307.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -49.69178625145058 (T) = (0 -9823.030584337912) / Math.Sqrt((13684.933875758907 / (16)) + (11870.461788617115 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.2193576319615305 = (8055.90609912861 - 9823.030584337912) / 8055.90609912861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 107.06901920351474 > 80.04532820996762.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -84.11993418870563 (T) = (0 -107003.00983265728) / Math.Sqrt((1889681.8862564708 / (16)) + (657636.8054703587 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.41837803668540674 = (75440.40239280039 - 107003.00983265728) / 75440.40239280039 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 255.6604948038642 > 202.7781252893519.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -55.884041599580996 (T) = (0 -257282.8595864423) / Math.Sqrt((17527228.73272934 / (16)) + (5296248.884157102 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.3254849925332067 = (194104.69453504335 - 257282.8595864423) / 194104.69453504335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 246.97714739583333 > 191.9066593633507.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -79.49440591702994 (T) = (0 -247251.84519395084) / Math.Sqrt((6432715.778143587 / (16)) + (7883845.747385901 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.35857370475830097 = (181993.69259685365 - 247251.84519395084) / 181993.69259685365 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.34994724620617 > 79.97152693146222.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -95.71100457062592 (T) = (0 -106838.53215812317) / Math.Sqrt((1433721.5267505643 / (16)) + (565019.9403261874 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.42023018565369047 = (75226.20856628851 - 106838.53215812317) / 75226.20856628851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 375.1153286564626 > 240.0623526177536.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -125.36743826940456 (T) = (0 -369071.9137320468) / Math.Sqrt((4930868.945078825 / (16)) + (27045143.07651314 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.6218423048581114 = (227563.37815736988 - 369071.9137320468) / 227563.37815736988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 239.45479810606062 > 183.12525014583332.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -97.41192003417274 (T) = (0 -239232.30683482942) / Math.Sqrt((3655556.8368771435 / (16)) + (5852488.39539498 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -0.36859700198270245 = (174801.1331957112 - 239232.30683482942) / 174801.1331957112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6393245584821425 > 1.5828857285975741.
IsChangePoint: Marked as a change because one of 7/9/2022 7:00:36 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -11.147015135883505 (T) = (0 -1616657.4601266307) / Math.Sqrt((618005147.3029522 / (16)) + (606339042.050049 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.05619062075535846 = (1530649.324428238 - 1616657.4601266307) / 1530649.324428238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetFullPathForReallyLongPath - Duration of single invocation 64.10 μs 71.33 μs 1.11 0.05 False
GetFullPathNoRedundantSegments - Duration of single invocation 7.21 μs 8.01 μs 1.11 0.05 False
GetFullPathForLegacyLength - Duration of single invocation 13.00 μs 14.53 μs 1.12 0.04 False
GetFullPathWithRedundantSegments - Duration of single invocation 10.96 μs 11.82 μs 1.08 0.03 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetFullPathForReallyLongPath

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.33181898376851 > 67.29567254994876.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -26.201125767693902 (T) = (0 -71819.9541602832) / Math.Sqrt((845986.9978698221 / (16)) + (729617.4838773188 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.11347366253392877 = (64500.81090992556 - 71819.9541602832) / 64500.81090992556 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.008552339505947 > 7.602225929278474.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -25.008512932465056 (T) = (0 -8011.652923727094) / Math.Sqrt((1743.1484384643 / (16)) + (22220.493996155907 / (27))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (27) - 2, .025) and -0.10532914309681424 = (7248.205635183695 - 8011.652923727094) / 7248.205635183695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetFullPathForLegacyLength

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.525201482243176 > 13.666669895151434.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -22.675526623174278 (T) = (0 -14621.825364223743) / Math.Sqrt((20994.35215592668 / (16)) + (90322.57932975903 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -0.11505132087481773 = (13113.141153675451 - 14621.825364223743) / 13113.141153675451 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Path.GetFullPathWithRedundantSegments

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.820210115888662 > 11.528509021825885.
IsChangePoint: Marked as a change because one of 7/9/2022 7:00:36 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -6.354699691227201 (T) = (0 -11978.090185470328) / Math.Sqrt((67929.72604309126 / (16)) + (423554.3128590881 / (25))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (25) - 2, .025) and -0.0836861748104038 = (11053.09864044907 - 11978.090185470328) / 11053.09864044907 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfAnyValues - Duration of single invocation 2.29 μs 6.92 μs 3.02 0.04 False
IndexOfValue - Duration of single invocation 1.94 μs 3.96 μs 2.05 0.04 False
LastIndexOfValue - Duration of single invocation 1.89 μs 4.05 μs 2.14 0.05 False
IndexOfAnyThreeValues - Duration of single invocation 2.81 μs 9.46 μs 3.37 0.03 False
IndexOfAnyTwoValues - Duration of single invocation 2.35 μs 6.83 μs 2.91 0.04 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.920429657407408 > 2.409322251034752.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -281.8578578403449 (T) = (0 -6846.502350038352) / Math.Sqrt((296.77918250410437 / (16)) + (6765.5404424673025 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.9765454451184528 = (2300.1504516810405 - 6846.502350038352) / 2300.1504516810405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfValue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.9627561285476864 > 2.038093223809507.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -110.26730625743754 (T) = (0 -3992.4318881347444) / Math.Sqrt((208.82634758767304 / (16)) + (9675.380269428444 / (29))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (29) - 2, .025) and -1.058685988309532 = (1939.310759778905 - 3992.4318881347444) / 1939.310759778905 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.0499471053791885 > 1.9835841480699752.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -166.1178827971298 (T) = (0 -3962.475363483021) / Math.Sqrt((58.50534023182592 / (16)) + (4242.5628608851885 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.0930638183627408 = (1893.1459847137348 - 3962.475363483021) / 1893.1459847137348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.456704100890509 > 2.894484188151439.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -244.32656806504244 (T) = (0 -9366.831033552136) / Math.Sqrt((470.43968420964467 / (16)) + (19612.250430398763 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -2.386155389120518 = (2766.214174236396 - 9366.831033552136) / 2766.214174236396 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.825602807613619 > 2.454142180541016.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -154.39008592939777 (T) = (0 -6905.459175266752) / Math.Sqrt((46.598888724408845 / (16)) + (24438.366682497654 / (28))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (28) - 2, .025) and -1.955239894453312 = (2336.6831194406936 - 6905.459175266752) / 2336.6831194406936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added CompliationMode=wasm PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Aug 23, 2022
@lewing
Copy link
Member

lewing commented Sep 15, 2022

@SamMonoRT has this one been triaged?

@lewing
Copy link
Member

lewing commented Sep 15, 2022

cc @vargaz @radekdoulik @marek-safar

@naricc
Copy link
Member

naricc commented Sep 15, 2022 •
edited
Loading

@lewing This looks like it is the same or a similar issue to the one I brought up a few weeks ago here: dotnet/runtime#74395

I am a little confused why I am seeing this again, because it has some of the same benchmarks, but also has more.

@naricc
Copy link
Member

naricc commented Sep 15, 2022

#7981 <-- This also seems to be from the same time. Reporting was down for a while there, but most likely the same underlying cause?

@lewing
Copy link
Member

lewing commented Sep 15, 2022

Yeah but this is the interpreter so very different code paths.

@lewing lewing mentioned this issue Sep 15, 2022
Closed
@radekdoulik radekdoulik mentioned this issue Sep 15, 2022
Closed
@radekdoulik
Copy link
Member

I put information I had around about that to the issue above. @vargaz could you please that a look at it?

@SamMonoRT
Copy link
Member

dotnet/runtime#75709

@jeffhandley jeffhandley mentioned this issue Sep 20, 2022
Merged
2 tasks
@naricc naricc removed the untriaged label Sep 28, 2022
@adamsitnik
Copy link
Member

The regressions are gone, the issue can be closed:

image

image

image

image

image

image

@jeffhandley
Copy link
Member

Excellent; thanks for posting these results, @adamsitnik.

@DrewScoggins neither Adam nor I has permission to close this issue; can you close it please?

@lewing
Copy link
Member

lewing commented Sep 28, 2022

nor I

@naricc naricc closed this as completed Sep 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-wasm arch-x64 branch-refs/heads/main kind-micro os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lewing @jeffhandley @radekdoulik @adamsitnik @naricc @SamMonoRT