Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

[performanceautofiler]

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	d7c8bc10ac00fe330daa633f8a2b16ba208e6e56
Compare	44f1c1ec1eed7662d47dfbbf506d55d0152029eb
Diff	Diff

Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
NewCustomConverter - Duration of single invocation	40.53 μs	46.71 μs	1.15	0.21	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.71231816877153 > 43.88696154360346.
IsChangePoint: Marked as a change because one of 6/22/2022 11:10:17 PM, 7/16/2022 2:59:06 AM, 7/29/2022 4:11:34 AM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM.
IsRegressionStdDev: Marked as regression because -7.504475221869677 (T) = (0 -44920.93830592109) / Math.Sqrt((1436821.783873055 / (23)) + (2216765.2470964855 / (22))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (22) - 2, .025) and -0.07238090143156774 = (41888.97643174564 - 44920.93830592109) / 41888.97643174564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	d7c8bc10ac00fe330daa633f8a2b16ba208e6e56
Compare	44f1c1ec1eed7662d47dfbbf506d55d0152029eb
Diff	Diff

Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
NewCustomConverter - Duration of single invocation	40.53 μs	46.71 μs	1.15	0.21	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.71231816877153 > 43.88696154360346.
IsChangePoint: Marked as a change because one of 6/22/2022 11:10:17 PM, 7/16/2022 2:59:06 AM, 7/29/2022 4:11:34 AM, 8/4/2022 5:17:15 AM falls between 7/26/2022 2:47:26 PM and 8/4/2022 5:17:15 AM.
IsRegressionStdDev: Marked as regression because -7.504475221869677 (T) = (0 -44920.93830592109) / Math.Sqrt((1436821.783873055 / (23)) + (2216765.2470964855 / (22))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (22) - 2, .025) and -0.07238090143156774 = (41888.97643174564 - 44920.93830592109) / 41888.97643174564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	EgorBo
Labels:	area-System.Text.Json
Milestone:	-

[EgorBo]

Likely caused by #72228 cc @eiriktsarpalis

[eiriktsarpalis]

Guessing you mean #72937? That PR is outside the reported commit range.

[EgorBo]

@eiriktsarpalis yeah, I meant the one which is outside, it seems we didn't file an issue for it, see "global history" https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_ubuntu%2020.04/System.Text.Json.Serialization.Tests.ColdStartSerialization(SimpleStructWithProperties).NewCustomConverter.html

[eiriktsarpalis]

Got it. That particular PR changes how converters are resolved for properties (they are now resolved lazily via the metadata cache) which may or may not result in redundant calculations being made.

Given that the particular benchmark measures serialization with metadata being regenerated from scratch, unless regressions have been recorded in other benchmarks, I don't believe this is blocking release. We should still investigate in case there are any obvious fixes we could apply, but otherwise we could just move it to 8.0.0 -- the particular benchmark still records a net improvement compared to .NET 6 due to the changes in #64646.

[eiriktsarpalis]

Should be investigated in unison with #71392, it concerns the same benchmark that regressed as part of the same feature.