System.Text.Json Reference Loop Handling

[MoienTajik]

One of the key features of JSON.NET serialization was the ReferenceLoopHandling which gives the ability to ignore the reference loops likes this :

public class Employee
{
    public string Name { get; set; }

    public Employee Manager { get; set; }
}

private static void Main()
{
    var joe = new Employee { Name = "Joe - User" };
    var mike = new Employee { Name = "Mike - Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonConvert.SerializeObject(joe, new JsonSerializerSettings
    {
        Formatting = Formatting.Indented,
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
    });
	
    Console.WriteLine(json);
}

And it produces the appropriate result :

{
  "Name": "Joe User",
  "Manager": {
    "Name": "Mike Manager"
  }
}

However, I couldn't find such a feature in System.Text.JSON, And when I've tried to Serialize the same object with JsonSerializer :

private static void Main()
{
    var joe = new Employee { Name = "Joe User" };
    var mike = new Employee { Name = "Mike Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonSerializer.ToString(joe, new JsonSerializerOptions
    {
        WriterOptions = new JsonWriterOptions
        {
            Indented = true,
        }
    });

    Console.WriteLine(json);
}

I've got this exception :
System.InvalidOperationException: 'CurrentDepth (1000) is equal to or larger than the maximum allowed depth of 1000. Cannot write the next JSON object or array.'

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

[xsoheilalizadeh]

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET. If we decided to replace the Json.NET with the new System.Text.Json we should consider this feature as soon as it is possible.

[ahsonkhan]

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

This is a known limitation of System.Text.Json (we do not have this feature yet) and we have little runway left to design and implement this for 3.0. We plan to add this for vNext, for sure (there are quite a few features/capabilities in Json.NET that wouldn't work in System.Text.Json and given the constraints, we had to prioritize what made it in).

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET

Given we detect and throw for self-reference models, how could it cause bugs?

[ahsonkhan]

cc @steveharter

[AmrAlSayed0]

Would this API be good?
This new property should be added to the JsonWriterOptions struct.

public struct JsonWriterOptions
{
    public ReferenceLoopHandlingOption ReferenceLoopHandling { get; set; }
}

public struct ReferenceLoopHandlingOption
{
    //A static member for easy access to pre-defined options.
   // This causes it to ignore all looped refrences.
    public static ReferenceLoopHandlingIgnore = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Ignore );
   //This causes the writer to output the looped reference only once after the first time.
   // {
   //   "Name": "Joe User",
   //   "Manager": {
   //     "Name": "Mike Manager",
   //     "Manager": {
   //       "Name":"Mike Manager"
   //     }
   //   }
   // }
    public static ReferenceLoopHandlingOnce = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Once );
    // Current (and default?) behavior.
    public static ReferenceLoopHandlingAll = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.All );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling );
    // Also there is the option to specify how many times do you want the writer to write looped refrences (after the first one)
    // new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Many , 3 );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling , int loopOutputTimes );
}

public enum ReferenceLoopHandling
{
    Ignore ,
    Once ,
    Many ,
    All
}

[ahsonkhan]

From @josundt (https://github.com/dotnet/corefx/issues/40045#issue-477089090):

Object/collection graphs containing circular references (or multiple references to the same entity), can be handled in a few different ways when serializing/deserializing to/from JSON.

Best practice is naturally to avoid graphs with circular references when using JSON serialization, but techniques to deal with it exist, and some times it may be needed. We use it a few places in our software.

NewtonSoft.Json supports this, and I guess it is a goal for System.Text.Json to make the feature gap with NewtonSoft as small possible.

During serialization, when an object/collection that has already been serialized is referenced for the second time or more, the common technique I've seen is to replace the repeated objects/collection in the JSON with a

{ "$ref": "some-sort-of-pointer" }

Some use JSONPath to reference the object/array serialized previously in the JSON document.

NewtonSoft.Json uses a different techique with specific serialization settings: It adds an extra "$id" property to all serialized objects in the JSON document, and the "$ref" value is the "$id" property of the object/array serialized previously.

Example using NewtonSoft.Json:

using System;
using System.Collections.Generic;
using Newtonsoft.Json;

namespace CircularSerialization
{
    public class Bike
    {
        public List<Tire> Tires { get; set; } = new List<Tire>();
    }

    public class Tire
    {
        public Bike Bike { get; set; }
    }

    static class Program
    {
        static void Main()
        {
            var bike = new Bike();
            bike.Tires.Add(new Tire { Bike = bike });
            bike.Tires.Add(new Tire { Bike = bike });

            var settings = new JsonSerializerSettings
            {
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore
            };
            var serialized = JsonConvert.SerializeObject(bike, settings);

            Console.Write(serialized);
            // {"$id":"1","Tires":[{"$id":"2","Bike":{"$ref":"1"}},{"$id":"3","Bike":{"$ref":"1"}}]}
        }
    }
}

I guess that if System.Text.Json.JsonSerializer will support circular/multiple references, some more memory allocation will be required during serialization/deserialization when the setting is switched on (so such a setting should naturally be off by default).

Is there a plan to support options for handling circular references in JsonSerializerSettings?
And if so, will you support the "$id" technique or the JSONPath techique, or maybe support both?

Reference:
https://github.com/douglascrockford/JSON-js/blob/master/cycle.js

[euclid47]

Will the enhancement be released in 3.1?

[davidnmbond]

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

[MagicAndre1981]

Will the enhancement be released in 3.1?

Milestone is set to 5, so no it will be not part of 3.1 and so this is useless at all.

[julienGrd]

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

You can for the moment switch to newtonsoft, im facing the same problem and i use this class extensions (adapt the settings to your needs)

public static class HttpClientExtension
    {
        public static async Task<T> PostJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object postData)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPostData = JsonConvert.SerializeObject(postData, Settings);

            HttpContent body = new StringContent(stringPostData, Encoding.UTF8, "application/json");
            var response = await sender.PostAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }


        public static async Task<T> GetJsonCustomAsync<T>(this HttpClient sender, string requestUrl)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            var response = await sender.GetAsync(requestUrl);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        public static async Task<T> PutJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object putData)
            where T : new()
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPutData = JsonConvert.SerializeObject(putData, Settings);

            HttpContent body = new StringContent(stringPutData, Encoding.UTF8, "application/json");
            var response = await sender.PutAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        private static JsonSerializerSettings Settings = new JsonSerializerSettings()
        {
            DateParseHandling = DateParseHandling.None,
            DateTimeZoneHandling = DateTimeZoneHandling.Unspecified
        };
    }

[ahsonkhan]

Folks on this thread who are requesting this feature, do you need it for deserialization as well as serialization? Most of the examples here are about serialization.

Also, how heavily (if at all) do you rely on Newtonsoft.Json's MetadataPropertyHandling.ReadAhead feature (to enable out-of-order metadata property support):
https://www.newtonsoft.com/json/help/html/DeserializeMetadataPropertyHandling.htm

I'd be interested in seeing current usages.

Also cc @ajcvickers, @Andrzej-W

[Andrzej-W]

@ahsonkhan, people serialize objects to deserialize them later - we need support for loops in both of them. Now when we have Blazor Wasm (officially it will be released in May 2020) this is a typical scenario:

• we have ASP.NET Web API service which is used to read some data from database and serialize it,
• we read this response in Blazor and have to deserialize it.

Databases are usually designed in such a way that we can read child items when we have parent item and we can read parent item when we have one of its children. As a direct consequence we have bidirectional links between POCO classes used in Entity Framework. Simple example: in InvoiceHeader class we have a property with list of InvoiceLines and in every InvoiceLine we have a reference to InvoiceHeader. This is similar to Bike and Tire example in one of the posts above.

In my opinion this is such a basic requirement that it have to be implemented as soon as possible. Without support for reference loops it will be very hard to write any real world Blazor application or any other application which have to serialize and deserialize object graphs used in Entity Framework.

[ajcvickers]

@ahsonkhan Handling of cycles is required for serializing the results from EF (or any other OR/M) for the reasons stated by @Andrzej-W. However, I don't know which specific flags from Newtonsoft.Json map to this.

[BreyerW]

Handling loops on both serialize and deserialize is required to be useful on many complex scenarios (EF was good example, i also use it privately). Best if you also provide something akin to ReferenceResolver like json.net do, because not everyone needs full ref loops and sometimes they want to restrict it to types that have guid as an example.

[jozkee]

people serialize objects to deserialize them later - we need support for loops in both of them

I agree, having a way to support loops and the ability to round-trip them is a must; to address that, System.Text.Json should implement something similar to Json.Net's PreserveReferencesHandling.All and MetadataPropertyHandling.Default.

With that said, is there any value left in including a feature similar to ReferenceLoopHandling.Ignore?

[Darlingtone]

I don't think System.Text.Json should have be released nor made the default handler for ASP.Net without any form of Cycles (Reference Loop) handling, given how deeply dependent ASP.Net or any modern .Net application for that matter depends on EF.

Using EF for any real world application you are most likely (99.99% likely) to require Reference Loop Handling with serialization/deserialization, so am wondering how useful System.Text.Json is really is at this stage.

Secondly, for a truly drop in place replacement for Json.Net I would have expected that the default JsonSerializationOptions match as closely as possible to Json.Net defaults

[jozkee]

dotnet/corefx#41002 is a work in progress where I am shaping an API to provide support for preserving references and handling loops.

Instead of having two options as in Json.Net I am exposing just one while discarding forcing serialization and preservation granularity which are not as important to have on a first effort and also to avoid overlapping which may cause trivial behaviors.

The next table show the combination of ReferenceLoopHandling and PreserveReferencesHandling and how to get its equivalent on System.Text.Json's ReferenceHandling:

RLH\PRH	None	All	Objects	Arrays
Error	Default	Not supported	Not supported	Not supported
Ignore	Ignore	Not supported	Not supported	Not supported
Serialize	Not supported	Preserve	Not supported	Not supported

This new ReferenceHandling option will also be involved into Deserialization by enabling metadata reading when is set to Preserve.

All suggestions are welcome.

[josundt]

@jozkee: This covers my personal needs, but I guess the option to preserve only non-collection references could be needed for some.

[tnlthanzeel]

is this issue solved?

[MagicAndre1981]

is this issue solved?

no, still open until .NET 5 in next year.

[tnlthanzeel]

thanks for the reply sir. is there any workaround for this?

[ildoc]

Don't use it to serialize results from EntityFramework or stick with NewtonSoft.Json until this will be closed

[tnlthanzeel]

@ildoc , thanks sir

[paulovila]

thanks for the reply sir. is there any workaround for this?

Sure, see my previous comment. It was not ready for netcore3.1 at the time I was looking at it, but surely they’ll align the NuGet versions accordingly

[tnlthanzeel]

thanks for the reply sir. is there any workaround for this?

Sure, see my previous comment. It was not ready for netcore3.1 at the time I was looking at it, but surely they’ll align the NuGet versions accordingly

thanks

[DM2489]

We've just had to take a decision to back out of implementing a change to System.Text.Json purely because of this. We didn't realise it didn't have feature parity with JSON.NET until part way through implementing this, where more complexed EF core queries have child members.

[tnlthanzeel]

We've just had to take a decision to back out of implementing a change to System.Text.Json purely because of this. We didn't realise it didn't have feature parity with JSON.NET until part way through implementing this, where more complexed EF core queries have child members.

so wont we be getting a fix for the self referencing loop?

[schwietertj]

@tnlthanzeel It looks like the reference loop handling will be addressed in .net 5 which is slated for Nov 2020. Check out dotnet/corefx#41002.

[ChiefInnovator]

Where is this feature? We desperately need this.

[RichCrane]

@ChiefInnovator learn to read, this will be available in .NET 5 which will be released at end of 2020

You mean "learn to see". I read very well thank you.

[jozkee]

The preview version for ReferenceHandling feature is here.

• Download the System.Text.Json preview package version 5.0.0-alpha.1.20071.1 from dotnet5 gallery (through Visual Studio's NuGet client):
  https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json

• How to use it:
  Set ReferenceHandling.Preserve to ReferenceHandling property in JsonSerializerOptions.

var options = new JsonSerializerOptions
{
    ReferenceHandling = ReferenceHandling.Preserve
};

string json = JsonSerializer.Serialize(objectWithLoops, options);

Here's the spec doc including samples and notes about many scenarios.
If you want to add feedback, feel free to leave a comment in https://github.com/dotnet/corefx/issues/41002.

[asm2025]

I wish the new System.Text.Json.JsonSerializer hasn't been made the default serialization handler in .net core 3 unless it is production-ready. I don't see how can anyone depend on it without having such an essential feature :(.

[asm2025]

Also, there is currently no abstraction layer to make a smooth transition between Newtonsoft and the new System.Text.Json classes. JsonConvert and JsonSerializer, for example, are both static classes that will not play well with interfaces or dependency injection.

[riscie]

@asm2025 It is still really easy to use JSON.NET instead of System.Text.Json https://stackoverflow.com/questions/42290811/how-to-use-newtonsoft-json-as-default-in-asp-net-core-web-api

[asm2025]

@riscie Thank you.
I think I'm having the same issue as #13564.
The code keeps looping forever and tells me eventually the size of the response is too long. It works normally for simple classes but it seems to have an issue when there is a reference loop problem.

[asm2025]

@riscie You're right, it's easy to switch. I had conflicting versions of Newtonsoft. After I resolved the conflict, things seem to work fine again.
I made my own abstraction layer to switch between Newtonsoft and System.Text.Json just in case.

Thanks

[denmitchell]

I'm not sure if this will be helpful to anyone waiting for 5.0, but I created a custom serializer (SafeJsonSerializer) that prevents self-referencing loops by keeping track of object hashcodes and not serializing the same object twice in a descendant node. (Logically, this prevents serialization loops, but still serializes all of the objects.) I just spent a few minutes creating a JsonConverterFactory (and internal converter class) that calls SafeJsonSerializer. I am including the code below in case it is helpful to anyone. I have tested my serializer with several different object types and collections of varying complexity, but it may not work in every scenario. I haven't directly tested performance, but my unit/integration tests that use SafeJsonSerializer seem to run pretty fast. (I recently ported it from Newtonsoft to System.Text.Json.) At any rate, here are the converter and serializer classes:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace EDennis.AspNetCore.Base.Serialization {

    public class NonLoopingJsonConverter : JsonConverterFactory {
        public override bool CanConvert(Type typeToConvert) => true;

        public override JsonConverter CreateConverter(
            Type type,
            JsonSerializerOptions options) {

            JsonConverter converter = (JsonConverter)Activator.CreateInstance(
                typeof(NonLoopingConverterInner<>).MakeGenericType(
                    new Type[] { type }),
                BindingFlags.Instance | BindingFlags.Public,
                binder: null,
                args: new object[] { },
                culture: null);

            return converter;
        }

        private class NonLoopingConverterInner<TValue> :
            JsonConverter<TValue> {


            public override TValue Read(
                ref Utf8JsonReader reader, Type typeToConvert,
                JsonSerializerOptions options) {

                return JsonSerializer.Deserialize<TValue>(ref reader, options);
            }


            public override void Write(
                Utf8JsonWriter writer, TValue value,
                JsonSerializerOptions options /*options ignored*/) {

                SafeJsonSerializer.Serialize(value, writer);

            }
        }
    }


    public static class SafeJsonSerializer {

        static readonly MethodInfo SerializeEnumerableMethod = typeof(SafeJsonSerializer).GetMethod("SerializeEnumerable", BindingFlags.Static | BindingFlags.NonPublic);



        public static void Serialize<T>(T obj, Utf8JsonWriter jw) =>
            Serialize(obj, null, jw, new List<int> { });
        


        static void Serialize<T>(T obj, string propertyName, Utf8JsonWriter jw, List<int> hashCodes, bool isContainerType = false) {

            var jsonValueType = GetJsonValueKind(obj);
            if (jsonValueType == JsonValueKind.Array || jsonValueType == JsonValueKind.Object) {
                var hashCode = obj.GetHashCode();
                if (hashCodes.Contains(hashCode))
                    return;
                hashCodes.Add(hashCode);
            }

            if (isContainerType && jsonValueType == JsonValueKind.Null)
                return;

            if (propertyName != null) {
                jw.WritePropertyName(propertyName);
            }


            if (obj != null && obj.GetType().IsEnum) {
                jw.WriteStringValue(Enum.GetName(obj.GetType(), obj));
                return;
            }

            switch (jsonValueType) {
                case JsonValueKind.Undefined:
                    if (propertyName == null)
                        return;
                    jw.WriteNullValue();
                    break;
                case JsonValueKind.Null:
                    jw.WriteNullValue();
                    break;
                case JsonValueKind.True:
                    jw.WriteBooleanValue(true);
                    break;
                case JsonValueKind.False:
                    jw.WriteBooleanValue(false);
                    break;
                case JsonValueKind.String:
                    var result = JsonSerializer.Serialize(obj).Replace("\u0022", "");
                    jw.WriteStringValue(result);
                    break;
                case JsonValueKind.Number:
                    var num = Convert.ToDecimal(obj);
                    jw.WriteNumberValue(num);
                    break;
                case JsonValueKind.Array:
                    jw.WriteStartArray();
                    try {
                        List<object> list = new List<object>();
                        if (typeof(IEnumerable).IsAssignableFrom(obj.GetType())) {
                            IEnumerable items = (IEnumerable)obj;
                            foreach (var item in items)
                                list.Add(item);
                        } else if (obj is IEnumerable<object>)
                            foreach (var item in obj as IEnumerable<object>)
                                list.Add(item);
                        else if (obj is IOrderedEnumerable<object>)
                            foreach (var item in obj as IOrderedEnumerable<object>)
                                list.Add(item);

                        SerializeEnumerable(list, jw, hashCodes);
                    } catch {

                        //upon failure, use reflection and generic SerializeEnumerable method
                        Type[] args = obj.GetType().GetGenericArguments();
                        Type itemType = args[0];

                        MethodInfo genericM = SerializeEnumerableMethod.MakeGenericMethod(itemType);
                        genericM.Invoke(null, new object[] { obj, propertyName, jw, hashCodes });
                    }
                    jw.WriteEndArray();
                    break;
                case JsonValueKind.Object:
                    jw.WriteStartObject();
                    var type = obj.GetType();
                    if (type.IsIDictionary()) {
                        var dict = obj as IDictionary;
                        foreach (var key in dict.Keys)
                            Serialize(dict[key], key.ToString(), jw, hashCodes);
                    } else {
                        foreach (var prop in type.GetProperties().Where(t => t.DeclaringType.FullName != "System.Linq.Dynamic.Core.DynamicClass")) {
                            var containerType = IsContainerType(prop.PropertyType);
                            Serialize(prop.GetValue(obj), prop.Name, jw, hashCodes, containerType);
                        }
                    }
                    jw.WriteEndObject();
                    break;
                default:
                    return;
            }
        }

        static void SerializeEnumerable<T>(IEnumerable<T> obj, Utf8JsonWriter jw, List<int> hashCodes) {
            foreach (var item in obj)
                Serialize(item, null, jw, hashCodes);
        }


        static JsonValueKind GetJsonValueKind(object obj) {
            if (obj == null)
                return JsonValueKind.Null;
            var type = obj.GetType();
            if (type.IsArray)
                return JsonValueKind.Array;
            else if (type.IsIDictionary())
                return JsonValueKind.Object;
            else if (type.IsIEnumerable())
                return JsonValueKind.Array;
            else if (type.IsNumber())
                return JsonValueKind.Number;
            else if (type == typeof(bool)) {
                var bObj = (bool)obj;
                if (bObj)
                    return JsonValueKind.True;
                else
                    return JsonValueKind.False;
            } else if (type == typeof(string) ||
                type == typeof(DateTime) ||
                type == typeof(DateTimeOffset) ||
                type == typeof(TimeSpan) ||
                type.IsPrimitive
                )
                return JsonValueKind.String;
            else if ((type.GetProperties()?.Length ?? 0) > 0)
                return JsonValueKind.Object;
            else
                return JsonValueKind.Undefined;
        }


        static bool IsContainerType(Type type) {
            if (type.IsArray)
                return true;
            else if (type.IsIDictionary())
                return true;
            else if (type.IsIEnumerable())
                return true;
            else if (type.IsNumber())
                return false;
            else if (type == typeof(bool)) {
                return false;
            } else if (type == typeof(string) ||
                type == typeof(DateTime) ||
                type == typeof(DateTimeOffset) ||
                type == typeof(TimeSpan) ||
                type.IsPrimitive
                )
                return false;
            else if ((type.GetProperties()?.Length ?? 0) > 0)
                return true;
            else if (type == typeof(object))
                return true;
            else
                return false;
        }
    }


    internal static class TypeExtensions {
        internal static bool IsIEnumerable(this Type type) {
            return type != typeof(string) && type.GetInterfaces().Contains(typeof(IEnumerable));
        }
        internal static bool IsIDictionary(this Type type) {
            return
                type.GetInterfaces().Contains(typeof(IDictionary))
                || (type.IsGenericType && typeof(Dictionary<,>).IsAssignableFrom(type.GetGenericTypeDefinition()));
        }
        internal static bool IsNumber(this Type type) {
            return type == typeof(byte)
                || type == typeof(ushort)
                || type == typeof(short)
                || type == typeof(uint)
                || type == typeof(int)
                || type == typeof(ulong)
                || type == typeof(long)
                || type == typeof(decimal)
                || type == typeof(double)
                || type == typeof(float)
                ;
        }
    }
}

[fzan]

I agree, it's absolutely a must have for a serialization library.