JSON Serializer recommendations for 5.0

[steveharter]

Based on a prior internal review and feedback, putting various information together to discuss.

[marek-safar]

Link to GH hash instead of was measured in 5.0 master as of 3/27/2020

[steveharter]

I'll link that the next time I run the benchmarks.

[MichalStrehovsky]

What's the ReadyToRun setting for these measurements? If things were left at their defaults, System.Text.Json will be compiled with ReadyToRun because that's how we ship the library out of CoreFX and none of the other libraries will get that benefit. ReadyToRun has huge impact on startup time.

[steveharter]

cc @layomia

For the "overhead" status I provided later (e.g. 22ms overhead) those were done with a public 5.0 build.

[layomia]

The ReadyToRun setting for these benchmarks is false.

I ran them again with ReadyToRun set to true using a Release build of .NET 5 at dotnet/runtime@38c2d5513c - https://github.com/layomia/jsonconvertergenerator/blob/00e81d5411175c8874528f2837bfbf3319637426/run_benchmarks.py#L31-L32

Deserialize LoginViewModel

Test	Mean (us)	Ratio
System.Text.Json	24772	1.00
Json.NET	93081	3.76
Utf8Json	70570	2.85
Jil	93637	3.78

Serialize LoginViewModel

Test	Mean (us)	Ratio
System.Text.Json	22498	1.00
Json.NET	85379	3.79
Utf8Json	71207	3.17
Jil	70038	3.11

See more benchmarks for other POCOs. We have faster start-up perf than the other serializers. (Click to expand)

---

Deserialize LoginViewModel

Test	Mean (us)	Ratio
System.Text.Json	24772	1.00
Json.NET	93081	3.76
Utf8Json	70570	2.85
Jil	93637	3.78

Deserialize Location

Test	Mean (us)	Ratio
System.Text.Json	26710	1.00
Json.NET	84303	3.16
Utf8Json	70803	2.65
Jil	111646	4.18

Deserialize IndexViewModel

Test	Mean (us)	Ratio
System.Text.Json	36492	1.00
Json.NET	93758	2.57
Utf8Json	84988	2.33
Jil	131906	3.61

Deserialize MyEventsListerViewModel

Test	Mean (us)	Ratio
System.Text.Json	31599	1.00
Json.NET	101226	3.20
Utf8Json	84027	2.66
Jil	134111	4.24

Serialize LoginViewModel

Test	Mean (us)	Ratio
System.Text.Json	22498	1.00
Json.NET	85379	3.79
Utf8Json	71207	3.17
Jil	70038	3.11

Serialize Location

Test	Mean (us)	Ratio
System.Text.Json	23205	1.00
Json.NET	86671	3.74
Utf8Json	74519	3.21
Jil	71267	3.07

Serialize IndexViewModel

Test	Mean (us)	Ratio
System.Text.Json	25998	1.00
Json.NET	86792	3.34
Utf8Json	81306	3.13
Jil	92546	3.56

Serialize MyEventsListerViewModel

Test	Mean (us)	Ratio
System.Text.Json	31207	1.00
Json.NET	91737	2.94
Utf8Json	85461	2.74
Jil	100481	3.22

[MichalStrehovsky]

Are the Utf8Json numbers with https://github.com/neuecc/Utf8Json/tree/master/src/Utf8Json.UniversalCodeGenerator? It would be interesting to see how much pregeneration helps Utf8Json here.

[steveharter]

PTAL @layomia

[layomia]

No they aren't. I'll take a look and share numbers.

[davidfowl]

Excuse my ignorance here but what does this mean? Representing metadata in code (aka custom metadata) vs what storing it in the binary?

[steveharter]

Updating this to say minimize the amount of cached metadata by the serializer if we can just call into generated code to get the values or process them (such as writing out an escaped property directly instead of caching a JsonEncodedText.

[davidfowl]

Can we quantify this?

[steveharter]

I assume you mean minimizing RAM\heap usage? Or do you mean size-on-disk? I do not have numbers yet for either but can get some numbers depending on what you find useful.

I do know that with large numbers of POCOs + many properties + longer-than-you-think property names (which is common) the cached metadata becomes significant, although the memory used is probably less than what the CLR maintains for each Type. Also FWIW currently for every property the serializer tracks 3 variants of its name - which I am currently looking at removing 1.

[jkotas]

How can the reflection generated goo be ever faster than this? I think that this is the fastest you can get. I do not see what kind of magic can beat that.

[steveharter]

I was thinking if we have a fast way to call the getter\setter directly such as what you can do with MethodInfo.CreateDelegate but without having to use reflection to get the MethodInfo for a property.

If we could do that, it would be 2x as fast as the delegate "hop" approach as just as fast as IL Emit.

Here's pretty much every way to call a property getter:

Direct:2.
Reflection:831.
Manual loosely typed object delegate:49. **(what I have in prototype)**
Manual strongly typed delegate:28.
Generated static method which calls property:43.
MethodInfo getter:18. **(would be nice to be able to do something like this)**
Il getter:28.
Expression getter:11.

Benchmark source (click to expand)

using System;
using System.Diagnostics;
using System.Linq.Expressions;
using System.Reflection;
using System.Reflection.Emit;

namespace ConsoleApp20benchmark
{
    class Program
    {
        static void Main(string[] args)
        {
            const long Iterations = 10_000_000;

            var poco = new POCO();
            PropertyInfo pi = typeof(POCO).GetProperty("MyProperty");
            MethodInfo mi = pi.GetGetMethod();

            int value;

            var sw = new Stopwatch();

            {
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = poco.MyProperty;
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Direct:{sw.ElapsedMilliseconds}.");
            }

            {
                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = (int)mi.Invoke(poco, null);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Reflection:{sw.ElapsedMilliseconds}.");
            }

            {
                Func<object, int> ManualGetter = (obj) =>
                {
                    return ((POCO)obj).MyProperty;
                };

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = ManualGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Manual loosely typed object delegate:{sw.ElapsedMilliseconds}.");
            }

            {
                Func<POCO, int> ManualStronglyTypedGetter = (obj) =>
                {
                    return obj.MyProperty;
                };

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = ManualStronglyTypedGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Manual strongly typed delegate:{sw.ElapsedMilliseconds}.");
            }

            {
                Func<POCO, int> ManualStronglyTypedGetter = (obj) => POCO.CallMyPropertyGetter(obj);

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = ManualStronglyTypedGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Generated static method which calls property:{sw.ElapsedMilliseconds}.");
            }

            {
                // Unforunately, we must use MethodInfo to get the direct delegate.
                Func<POCO, int> DelegateGetter = (Func<POCO, int>)mi.CreateDelegate(typeof(Func<POCO, int>));

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = DelegateGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"MethodInfo getter:{sw.ElapsedMilliseconds}.");
            }

            {
                var dynamicMethod = new DynamicMethod(
                    mi.Name,
                    mi.ReturnType,
                    new[] { typeof(object) },
                    typeof(Program).Module,
                    skipVisibility: true);

                ILGenerator generator = dynamicMethod.GetILGenerator();
                generator.Emit(OpCodes.Ldarg_0);
                generator.Emit(OpCodes.Castclass, mi.DeclaringType); // to verify type
                generator.Emit(OpCodes.Callvirt, mi);
                generator.Emit(OpCodes.Ret);

                Func<POCO, int> IlGetter = (Func<POCO, int>)dynamicMethod.CreateDelegate(typeof(Func<,>).MakeGenericType(mi.DeclaringType, mi.ReturnType));

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = IlGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();
                
                Console.WriteLine($"Il getter:{sw.ElapsedMilliseconds}.");
            }

            {
                ParameterExpression parameter = Expression.Parameter(pi.DeclaringType, pi.Name);
                MemberExpression property = Expression.Property(parameter, pi);
                Func<POCO, int> ExpressionGetter = (Func<POCO, int>) Expression.Lambda(property, parameter).Compile();

                sw.Reset();
                sw.Start();
                for (long l = 0; l < Iterations; l++)
                {
                    value = ExpressionGetter(poco);
                    Debug.Assert(value == 42);
                }
                sw.Stop();

                Console.WriteLine($"Expression getter:{sw.ElapsedMilliseconds}.");
            }
        }


        class POCO
        {
            public int MyProperty => 42;

            internal static int CallMyPropertyGetter(POCO obj)
            {
                return obj.MyProperty;
            }
        }
    }
}

[jkotas]

Here's pretty much every way to call a property getter (except for expressions):

These numbers look suspect. I think you are seeing a lot of noise from tiered JIT, and other first-time initializations. I have added a big for-loop over the whole Main method. Here is the difference between 1st and 10th iteration (these number are on 5.0.100-preview.4.20212.3):

1st iteration:

Direct:3.
Reflection:1233.
Manual loosely typed object delegate:80.
Manual strongly typed delegate:52.
Generated static method which calls property:80.
MethodInfo getter:37.
Il getter:44.
Expression getter:24.

10th iteration:

Direct:3.
Reflection:1104.
Manual loosely typed object delegate:26.
Manual strongly typed delegate:23.
Generated static method which calls property:26.
MethodInfo getter:30.
Il getter:33.
Expression getter:20.

[jkotas]

The 10th iteration numbers say that direct call is super fast (it should actually get optimized out completely in your benchmark and so you are just measuring how fast one can count to 10_000_000), the reflection Invoke is slow, and the rest is pretty much the same - the differences are in the noise range.

[jkotas]

Also, the distribution would be different for something that returns value type: It would show that operating on objects has extra overhead due to boxing.

[jkotas]

For example, the expression getter should have by far the worst cold start characteristic because of it pulls in Expressions that are big expensive piece of code.

[steveharter]

Yes definitely the Emit and Expression approaches are out for cold start. Plus the System.Linq.Expressions assembly is very large if it needs to be pulled in.

Since today we use Emit I think we likely have two options:

1. For AOT, a code-gen'd delegate approach as shown in the doc's prototype will be fast and work with linkers.
2. For non-AOT a new reflection feature is needed that does not have cold start issues but still has fast steady-state performance.

[steveharter]

For non-AOT, until the "new reflection feature" is available, I suggest we continue to do what we do today which is use Emit where it is supported and use the slow Reflection.Invoke as a fallback.

[steveharter]

Ideally we have one approach for both AOT and non-AOT but until we get further along with the "new reflection feature" I'm not sure if that will be feasible.

We also need to determine the priority of linker support, and\or whether the new reflection feature could support the linker in some way.

[jkotas]

We also need to determine the priority of linker support

+1. I think there is no shared understanding of it today.

Would it make sense to simplify this to just two scenarios that we focus on?

• The dynamic mode that we have today
• The linker and AOT friendly mode

I understand you can have number of options in between, but I am worried that having many different options will be hard to explain.