Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

.NET6 System.Text.Json source-generator appears slower than .NET5 when doing Deserialization #61501

Closed
JPThorne opened this issue Nov 12, 2021 · 6 comments
Closed

.NET6 System.Text.Json source-generator appears slower than .NET5 when doing Deserialization #61501

JPThorne opened this issue Nov 12, 2021 · 6 comments
Labels
area-System.Text.Json tenet-performance Performance related issue untriaged New issue has not been triaged by the area owner

Comments

@JPThorne
Copy link

Description

Hello, I was taking a look at: https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/ and thought I'd give the new source-generator approach a try.

However, to my confusion a comparison benchmark run against net5.0 appears to be faster on de-serialization than the same thing on net6.0. The serialization is definitely faster though (yay).

So of course I am wondering if that is meant to be the case, what am I missing here? I'd at least expect de-serialization to be the same speed, not slower.

To reproduce this I put up 3 repos here, with a summary attached:

The results were generated by running dotnet run -c Release against each repo in turn. Any feedback would be much appreciated of course, not 100% sure this Issue is in the correct repo.

Configuration

  • Which version of .NET is the code running on?
  • .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  • .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  • .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT

  • What OS version, and what distro if applicable?
    Windows 10 Pro, 10.0.19043 Build 19043

  • What is the architecture (x64, x86, ARM, ARM64)?
    x64

  • If relevant, what are the specs of the machine?
    Processor Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz 1.80 GHz
    Installed RAM 16,0 GB (15,8 GB usable)
    System type 64-bit operating system, x64-based processor

Regression?

I don't think this is a regression per-se, or perhaps it is as de-serialization seems slower now?

Data

net3.1

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
  DefaultJob : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
Method Mean Error StdDev
Serialize 446.5 ns 6.73 ns 6.91 ns
Deserialize 979.0 ns 13.08 ns 12.23 ns

net5.0

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  DefaultJob : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
Method Mean Error StdDev
Serialize 401.7 ns 3.27 ns 2.73 ns
Deserialize 427.9 ns 4.12 ns 3.85 ns

net6.0

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  DefaultJob : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Method Mean Error StdDev
Serialize 245.8 ns 3.83 ns 3.40 ns
Deserialize 500.9 ns 6.63 ns 6.20 ns

Analysis

Some test repos I created:
@JPThorne JPThorne added the tenet-performance Performance related issue label Nov 12, 2021
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Nov 12, 2021
@ghost
Copy link

ghost commented Nov 12, 2021

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Hello, I was taking a look at: https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/ and thought I'd give the new source-generator approach a try.

However, to my confusion a comparison benchmark run against net5.0 appears to be faster on de-serialization than the same thing on net6.0. The serialization is definitely faster though (yay).

So of course I am wondering if that is meant to be the case, what am I missing here? I'd at least expect de-serialization to be the same speed, not slower.

To reproduce this I put up 3 repos here, with a summary attached:

The results were generated by running dotnet run -c Release against each repo in turn. Any feedback would be much appreciated of course, not 100% sure this Issue is in the correct repo.

Configuration

  • Which version of .NET is the code running on?
  • .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  • .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  • .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT

  • What OS version, and what distro if applicable?
    Windows 10 Pro, 10.0.19043 Build 19043

  • What is the architecture (x64, x86, ARM, ARM64)?
    x64

  • If relevant, what are the specs of the machine?
    Processor Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz 1.80 GHz
    Installed RAM 16,0 GB (15,8 GB usable)
    System type 64-bit operating system, x64-based processor

Regression?

I don't think this is a regression per-se, or perhaps it is as de-serialization seems slower now?

Data

net3.1

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
  DefaultJob : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
Method Mean Error StdDev
Serialize 446.5 ns 6.73 ns 6.91 ns
Deserialize 979.0 ns 13.08 ns 12.23 ns

net5.0

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  DefaultJob : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
Method Mean Error StdDev
Serialize 401.7 ns 3.27 ns 2.73 ns
Deserialize 427.9 ns 4.12 ns 3.85 ns

net6.0

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
Intel Core i5-8265U CPU 1.60GHz (Whiskey Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  DefaultJob : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Method Mean Error StdDev
Serialize 245.8 ns 3.83 ns 3.40 ns
Deserialize 500.9 ns 6.63 ns 6.20 ns

Analysis

Some test repos I created:

Author: JPThorne
Assignees: -
Labels:

area-System.Text.Json, tenet-performance, untriaged
Milestone: -

@eiriktsarpalis
Copy link
Member

Hi @JPThorne, I made a few changes to your net6 benchmark so that it's more deterministic and runs in all considered TFMs so it's easier to compare. My results are slightly different:

.NET Core 3.1

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i9-10900X CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100
  [Host]     : .NET Core 3.1.21 (CoreCLR 4.700.21.51404, CoreFX 4.700.21.51508), X64 RyuJIT
  DefaultJob : .NET Core 3.1.21 (CoreCLR 4.700.21.51404, CoreFX 4.700.21.51508), X64 RyuJIT
Method Mean Error StdDev
Serialize_Reflection 399.2 ns 7.18 ns 8.54 ns
Deserialize_Reflection 567.6 ns 11.31 ns 13.46 ns

.NET 5

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i9-10900X CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 5.0.12 (5.0.1221.52207), X64 RyuJIT
  DefaultJob : .NET 5.0.12 (5.0.1221.52207), X64 RyuJIT
Method Mean Error StdDev
Serialize_Reflection 348.8 ns 7.02 ns 8.36 ns
Deserialize_Reflection 490.6 ns 9.82 ns 11.69 ns

.NET 6

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i9-10900X CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  DefaultJob : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Method Mean Error StdDev
Serialize_Reflection 275.8 ns 5.23 ns 5.37 ns
Deserialize_Reflection 416.4 ns 7.21 ns 6.75 ns
Serialize_SourceGen 199.0 ns 4.04 ns 3.97 ns
Deserialize_SourceGen 317.7 ns 6.29 ns 5.57 ns

Note that the performance gains in sourcegen deserialization aren't as pronounced, but that is to be expected, fast-path deserialization has not been implemented yet (see #55043).

@JPThorne
Copy link
Author

Hi @eiriktsarpalis thanks so much for the quick feedback. And good idea on combining them all in one repo, much easier to compare. My local results are consistent with yours, in the combined repo.

I was curious though if I could get my standalone net6.0 repo to test faster than the net5.0 one, and if I don't essentially define the JsonSerializerOptions twice then it is in fact faster on my side as well.

Removing:

_options = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };

etc. as you did seems to make a difference.

Thanks.

@eiriktsarpalis
Copy link
Member

Removing PropertyNamingPolicy etc. as you did seems to make a difference.

Could you clarify? I removed it so that we could focus testing on the baseline. Do you have indication that the particular feature has regressed in 6?

@steveharter
Copy link
Member

@eiriktsarpalis the reflection test is not using the camel-case policy, while the source-gen is however both share the same JSON which is case-sensitive and Pascal-cased. This means the source-gen deserialize test is artificially much faster since none of the properties matched up and set.

If the reflection serializer is updated to use the camel-case policy then the deserialize test should be just as fast as the source-gen one (since there is no fast-path for source-gen + deserialize).

@eiriktsarpalis
Copy link
Member

eiriktsarpalis commented Nov 12, 2021 •
edited
Loading

I hadn't noticed that the sourcegen class contained custom configuration. I removed it in eiriktsarpalis/net6-json-test@70132dd which should provide an apples-to-apples comparison:

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i9-10900X CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100
  [Host]     : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  DefaultJob : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
Method Mean Error StdDev
Serialize_Reflection 309.0 ns 6.07 ns 6.23 ns
Deserialize_Reflection 439.2 ns 7.84 ns 7.34 ns
Serialize_SourceGen 214.7 ns 4.30 ns 6.30 ns
Deserialize_SourceGen 443.6 ns 6.18 ns 5.16 ns

@ghost ghost locked as resolved and limited conversation to collaborators Dec 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Text.Json tenet-performance Performance related issue untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JPThorne @eiriktsarpalis @steveharter