Proposal: Add mechanism to handle circular references when serializing

[jozkee]

See initial proposal with extended comments here:
https://github.com/dotnet/runtime/pull/354/files

See proposal extension for ReferenceResolver on #30820 (comment).

Rationale and Usage

Currently there is no mechanism to prevent infinite looping in circular objects nor to preserve references when using System.Text.Json.

Community is heavily requesting this feature since is consider by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though JSON specifiacation does not support reference loops by default. Therefore this will be shipped as an opt-in feature.

The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance.

Proposed API

namespace System.Text.Json
{
    public class JsonSerializerOptions
    {
        public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default;
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// This enumeration defines the various ways the <see cref="JsonSerializer"/> 
    /// can deal with references on serialization and deserialization.
    /// </summary>
    public enum ReferenceHandling
    {
        Default,
        Preserve
    }
}

EDIT:

We considered having ReferenceHandlig.Ignore but it was cut out of the final design due the lack of scenarios where you would really need Ignore over Preserve.

Although is not part of the shipping API, the samples and definitions of Ignore remain in this description for their informative value.

In depth

• Default:

  • On Serialize: Throw a JsonException when MaxDepth is exceeded, this may occur by either a Reference Loop or by passing a very deep object. This option will not affect the performance of the serializer.
  • On Deserialize: No effect.

• Ignore:

  • On Serialize: Ignores (skips writing) the property/element where the reference loop is detected.
  • On Deserialize: No effect.

• Preserve:

  • On Serialize: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array.
  • On Deserialize: While the other options show no effect on Deserialization, Preserve does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it.

Feature Parity (Examples of System.Text.Json vs Newtonsoft's Json.Net)

Having the following class:

class Employee 
{ 
    string Name { get; set; }
    Employee Manager { get; set; }
    List<Employee> Subordinates { get; set; }
}

Using Ignore on Serialize

On System.Text.Json:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Ignore
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "Name": "Angela",
    "Manager": {
        "Name": "Bob",
        "Subordinates": [] //Note how subordinates is empty due Angela is being ignored.
    }
}

Using Preserve on Serialize

On System.Text.Json:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        PreserveReferencesHandling = PreserveReferencesHandling.All
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "$id": "1",
    "Name": "Angela",
    "Manager": {
        "$id": "2",
        "Name": "Bob",
        "Subordinates": { //Note how the Subordinates' square braces are replaced with curly braces in order to include $id and $values properties, $values will now hold whatever value was meant for the Subordinates list.
            "$id": "3",
            "$values": [
                {  //Note how this object denotes reference to Angela that was previously serialized.
                    "$ref": "1"
                }
            ]
        }            
    }
}

Using Preserve on Deserialize

On System.Text.Json:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
    };

    Employee angela = JsonSerializer.Deserialize<Employee>(json, options);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

On Newtonsoft's Json.Net:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerSettings
    {
        MetadataPropertyHanding = MetadataPropertyHandling.Default //Json.Net reads metadata by default, just setting the option for ilustrative purposes.
    };

    Employee angela = JsonConvert.DeserializeObject<Employee>(json, settings);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

Notes:

1. MaxDepth validation will not be affected by ReferenceHandling.Ignore or ReferenceHandling.Preserve.
2. We are merging the Json.Net types ReferenceLoopHandling and PreserveReferencesHandling (we are also not including the granularity on this one) into one single enum; ReferenceHandling.
3. While Immutable types and System.Arrays can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference.
4. Value types, such as structs, will not be supported when Deserializing as well.
5. Additional features, such as Converter support, ReferenceResolver, JsonPropertyAttribute.IsReference and JsonPropertyAttribute.ReferenceLoopHandling, that build on top of ReferenceLoopHandling and PreserveReferencesHandling were considered but they will not be included in this first effort.
6. We are still looking for evidence that backs up supporting ReferenceHandling.Ignore.

Issues related:

• https://github.com/dotnet/corefx/issues/38579
• https://github.com/dotnet/corefx/issues/37786
• https://github.com/dotnet/corefx/issues/40045
• https://github.com/dotnet/corefx/issues/41139
• https://github.com/dotnet/corefx/issues/41288
• What is Newtonsoft.Json's ReferenceLoopHandling equivalent in System.Text.Json? aspnetcore#14497

[jozkee]

@ahsonkhan @JamesNK thoughts?

[JamesNK]

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

{
  "$id": "1",
  "Name": "Angela",
  "Subordinates": {
    "$id": "2",
    "$values": [
      {
        "$id": "3",
        "Name": "Bob",
        "Manager": {
            "$ref": "1"
        }
      }
    ]
  }
}

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

[JsonReferenceHandling(ReferenceHandling.Preserve)]

// to

[JsonProperty(ReferenceHandling = ReferenceHandling.Preserve)]

[jozkee]

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

Agree.

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

[ahsonkhan]

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

Correct, we have JsonPropertyNameAttribute
https://github.com/dotnet/corefx/blob/b41b09eadd5eb3f5575845a982f2c177c37f7ce7/src/System.Text.Json/ref/System.Text.Json.cs#L748-L753

cc @steveharter

[scalablecory]

Is this proposal compatible with Json.Net?

[scalablecory]

ReferenceHandling: byte

Why byte?

[JamesNK]

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Is there a way to override equality? Newtonsoft.Json has https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializerSettings_EqualityComparer.htm

[JamesNK]

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

What happens if $id isn't the first property?

[jozkee]

Is this proposal compatible with Json.Net?

Yes for the metadata semantics that I am trying to follow; no for a feature parity, however we are not tied to follow the first one, just come up with what is already there since I have not found value in going on a different direction.

Why byte?

I was following existing enums in the namespace that also inherits from byte, thing that is made to reduce their size in the Utf8JsonReader struct. but since the Serializer is a class, you are right, there is no much value in changing the enum's default value, I will fix that. Thanks.

[jozkee]

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Currently the implementation relies on default equality of Hashset/Dictionary, which uses the Object.GetHashCode() and Object.Equals(Object obj).
If someone wants to provide its own equals mechanism, he can override these methods and/or implement IEquatable on its class.

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

Json.Net throws when finds that there are properties other than $ref in a JSON reference object, I was planning on sticking to that behavior.
I see no point in allowing other properties to co-exist with a $ref, unless the user relies on the $ref property for another purpose; for that maybe in a future we could add a mechanism to disable this functionality similar to MetadataPropertyHandling.Ignore

[steveharter]

@JamesNK on "Ignore" - is this mode necessary and what scenarios would prefer that over "Preserve"?

"Ignore" has potnential inconsistentcy w.r.t. what is serialized vs. what is ignored which would make me question using it in certain scenarios including:

• Serialization order currently depends on reflection order which is not deterministic, so if property A and B both serialize the same reference, sometimes A will be serialized and sometimes B will be serialized.
• The root of the graph may change causing a given type to sometimes serialize a reference, and other times not.

In many scenarios the [JsonIgnore] could be used instead to prevent cycles which has the advantage of being deterministic.

However if:

• "Throws" isn't desired
• and "Preserve" isn't desired because the metadata ($id $ref) or other reasons (performance?)
• and [JsonIgnore] can't be used (perhaps attributes can't be used)
• and potential non-deterministic behavior is OK

then "Ignore" makes sense.

[terrajobst]

Video

We seem to lean towards not having Ignore -- it results in payloads that nobody can reason about. The right fix for those scenarios to modify the C# types to exclude back pointers from serialization.

We'd like to see more detailed spec for the behavior, specifically error cases.

[JamesNK]

Ok, I don't feel strongly that ignore is necessary. And it should never be the default setting so you can always add it in the future if desired.

[paulovila]

Hi @jozkee , is there a way to try PreserveReferencesHandling, like in a nuget preview?

[jozkee]

@paulovila this feature is still a work in progress, once is finished and merged into our master branch you can try it out by installing the latest bits of Net core.

[los93sol]

This is extremely disappointing, it was called out by numerous people while 3.0 was in preview as a complete blocker for a lot of people to use the new serializer and benefit from it. It’s even more disappointing this still isn’t being prioritized properly. It seems like by the time this gets addressed people will have just abandoned the new serializer altogether and likely won’t return to it.

[ahsonkhan]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

[los93sol]

IMO this is a major breaking change and it actually roadblocks a lot of people from following the recommended serializer setup from the docs. I’m frustrated because despite this being called out in preview there’s no plan to address in 3.0 or even 3.1 that I’ve seen. Waiting for 5.0 is way too far out on the roadmap and why I said it still isn’t being prioritized properly.

[mythrz]

Edit: Deleted the other comments. Are meaningless!! I was following the wrong crumbles...
In my case, I had an additional Method (garbage/test method...) inside the class that I wanted to serialize... mb, the essential is working.

[doogdeb]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

[mythrz]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

[doogdeb]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

I understand that, but you mention that this change is earmarked for 5.0, which I'm assuming is .Net 5.0, which is a year away.

[nickcoad]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

"Next year, around this time" is just another way of saying "a year away"...

[ahsonkhan]

Isn't 5.0 a year away?

I understand that, but you mention that this change is earmarked for 5.0, which I'm assuming is .Net 5.0, which is a year away.

"Next year, around this time" is just another way of saying "a year away"...

Yes, this feature is for 5.0 (i.e. master), which is planned to ship near end of next year (hence the milestone). 3.1 is already done. That said, S.T.Json also ships as a NuGet package so once the feature is done, you could consider using a preview package on 3.1 before 5.0 officially ships.

[paulovila]

It's complicated to depend on the newtonsoft serialisation patch because It is not being updated at the same pace as the SignalR patch is, and that causes that we can’t upgrade to the intermediate versions that you provide. 🤷‍♂️

[ahsonkhan]

@paulovila - I am sorry, but I don't understand what you mean. What target framework are you using in your project?

• What do you mean by "the newtonsoft serialisation patch"? Why is the versioning/shipping of Newtonsoft affect your use of System.Text.Json?
• What do you mean by "SignalR patch"? SignalR generally ships at the same time as .net core, asp.net core, as far as I know.

and that causes that we can’t upgrade to the intermediate versions that you provide.

Let's say you have a .NET Core/Asp.Net Core 3.1 application. The built-in System.Text.Json library that shipped with 3.1 doesn't contain this reference handling feature. However, if you are fine with referencing preview packages, you can add a PackageReference to the latest 5.0 nightly build of the System.Text.Json NuGet package and get that feature (once its implemented/merged into master).

[paulovila]

@ahsonkhan the aforementioned patches are:

<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="3.0.0" />
<PackageReference Include="Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson" Version="3.0.0" />

with that you can circumvent the circular references as follows:

using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
...
services.AddMvc()
    .AddNewtonsoftJson(options =>
    {
        options.SerializerSettings.NullValueHandling = NullValueHandling.Ignore;
        options.SerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
    })
    .SetCompatibilityVersion(CompatibilityVersion.Version_3_0);

services.AddSignalR().AddNewtonsoftJsonProtocol(options =>
{
    options.PayloadSerializerSettings.NullValueHandling = NullValueHandling.Ignore;
    options.PayloadSerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
});

Is this patch strategy the right way to go? I can see now that there has been some alignment in the nuget preview versions since I tried version 3.0.0.0.

[AlbertoPa]

We considered having ReferenceHandlig.Ignore but it was cut out of the final design due the lack of scenarios where you would really need Ignore over Preserve.

Here: https://docs.microsoft.com/en-us/ef/core/querying/related-data/serialization Literally, straight out of the official MS documentation.

[BrunoBlanes]

@AlbertoPa I think for that example you can set the ReferenceHandler property to Preserve.

[AlbertoPa]

@BrunoBlanes I have tried it but it does not seem to work if there are many-to-many relationships. Currently I have two workarounds:

1. Annotate the two navigation properties in the join entity with [JsonIgnore]

or

2. Use Newtonsoft.Json with the

options.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;

option on the server. In a Blazor app, the System.Text.Json deserializer seems to be OK with this trick so far.

[BrunoBlanes]

@AlbertoPa Could it be that you may be doing something wrong? I got it working fine o Blazor, I just had an issue with naming policies, but it was my fault.

[AlbertoPa]

@BrunoBlanes certainly possible, but I don't see where. I have made a basic case where the join entity has only id's and navigation properties. If the EF query populates the IList of join entities and the corresponding navigation properties (meaning Include and ThenInclude), I can't get it to work properly even if I specify the Preserve option.

What was the issue with naming policies? I am using the standard naming convention in EF Core.

[BrunoBlanes]

Forgot to set the naming policies: #45639

What error are you getting?

[AlbertoPa]

This is what I get:

Unhandled exception rendering component: The JSON value could not be converted to System.Collections.Generic.List1[Solver]. Path: $.items | LineNumber: 0 | BytePositionInLine: 63.`

I haven't tried to play with naming policies. Will give it a try.

[BrunoBlanes]

Check if the JSON you are trying to deserialize is an actual list, then check your JSON for all of Solver's property types to see if they match your C# class, (integers can't be in quotes, bool need to be true or false, etc.)

[ahsonkhan]

@AlbertoPa also if you could share a minimal standalone repro (potentially with data sanitization), with the JSON you are processing and the .NET object model used for serialization, folks on the thread might be able to help investigate.

[ryantgray]

It appears I am having the same issue as AlbertoPa. I've posted the issue several places and so far have received no solution. I don't want to clutter up this post anymore than necessary so I'm wondering if we can start here...

I have a Blazor WebAssembly application that so far has been working fine in everything I have implemented so far. The last thing I did was created a couple new classes in the Shared project that represent a parent/child relationship and my parent has a property for a collection of child objects and the child class has a property referencing its parent. I am also using EF. Once I implemented all of this and tried to run the app and browse to the page i got the following error at which point I fired up the Googler in search or a solution:

• JsonException: A possible object cycle was detected which is not supported. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32.

So my first question would be, is this expected behavior for a default implementation of a Blazor WebAssembly application? If so, can someone please post what the specific steps are to resolve this issue?

Based on what I've read so far, it looks to me like this is the expected behavior and that a solution was implemented for it which is what this post represents. It looks to me like it states a solution was implemented for it as part of .NET 5 and that the solution that I would want would be to set ReferenceHandling = ReferenceHandling.Preserve.

If wanted, i can go into the details of what I have tried so far, etc. and give as much detail as possible but the approach I would prefer taking is someone just posting the specific steps of what needs to be done.

FYI, when i tried implementing the code changes that it looked to me like i needed to make based on this post one or two other sites I read all that did was instead of getting the error I mentioned above, I then got the error that AlbertPa mentioned. I would not THINK this would be related to naming policies since, as i mentioned, everything else up to this point had been working fine unless the issues resolved by naming policies would only come into play in my type situation when there are circular references involved. Thanks.

[BrunoBlanes]

So my first question would be, is this expected behavior for a default implementation of a Blazor WebAssembly application? If so, can someone please post what the specific steps are to resolve this issue?

@ryantgray That is the default behavior for System.Text.Json, and not only for Blazor WebAssembly. Since you said you've posted this issue on several places, do you happen to have an issue number with some proper debugging info?

[AlbertoPa]

I could somewhat address the issue with the change in naming convention you suggested. This however resulted in larger payload than with Ignore in Newtonsoft.

Also, while not a bug, there is no way to have global options in the client, so it means changing the whole code base. Definitely not making Text.Json a plug in replacement, so for now I'll stay with my hybrid solution, using Newtonsoft on the server.

[BrunoBlanes]

I could somewhat address the issue with the change in naming convention you suggested. This however resulted in larger payload than with Ignore in Newtonsoft.

Apparently there are some risks of data loss with ReferenceHandling.Ignore so they thought better not to implement it, since the only benefit would be a payload reduction.

Also, while not a bug, there is no way to have global options in the client, so it means changing the whole code base. Definitely not making Text.Json a plug in replacement, so for now I'll stay with my hybrid solution, using Newtonsoft on the server.

For this I think you could create an extension method.

[AlbertoPa]

Yes, about extension methods.

The rest has been standard practice for years without issues (it's literally recommended by EF doc). I'd say leave to the developer and implement the functionality. Until then, Text.Json isn't a replacement for Newtonsoft without extra work, and the benefit of performance may not be of interest due to the extra work needed. My two cents.

[BrunoBlanes]

If you are not happy you can always not use S.T.Json, it is completely optional.

[AlbertoPa]

It is not a question of happiness, but of functionality. This is telling users to implement custom code for basic functionality, and I'd say having a global way to pass options to a serializer/deserializer also on the client is not asking too much for a framework which heavily relies on those operations.

Frankly speaking, looking at all the threads and on StackOverflow here on this issue, there has been a lot of energy wasted arguing that "Ignore is not needed". Probably the same energy could have been channelled into implementing the functionality, and resolve the issue.

[ryantgray]

@BrunoBlanes Here are the issue numbers I have read and/or commented on regarding this issue:

#29900

dotnet/aspnetcore#28286

#42584

It looked to me like the fix for the problem was to update System.Net.Http.Json to version 5.0.0 in the client project and then
implement one of the following two code changes (as described in the referenced articles above) in the ConfigureServices method
of Startup.cs in the Server project:

services.AddControllersWithViews().AddJsonOptions(options =>
{
options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
});

or

services.AddControllersWithViews(options =>
{
options.OutputFormatters.RemoveType();
options.OutputFormatters.Add(new SystemTextJsonOutputFormatter(new JsonSerializerOptions(JsonSerializerDefaults.Web)
{
ReferenceHandler = ReferenceHandler.Preserve
}));
});

Am i on the right track here? Is there anything else that needs to be done? Do i also need to update my Server project
to change it from targeting .NET Core 3.1 to targeting .NET 5.0?

[butler1233]

@ryantgray I thionk you are on the right track. You shouldn't need to update to .Net 5, as long as you're running STJ 5 then you should be good. (I believe the ReferenceHandler flag didn't work at all in v3) Preserve is an excellent choice for retaining the full object tree, although it absolutely has drawbacks, as well as advantages:

Drawbacks

1. It puts a load of additional metadata into the JSON, which as far as I'm aware only STJ knows how to parse correctly.
2. It kinda breaks the original data structure - This is related to the metadata, but a lot of the actual data gets moved around and it's not as human readable as normally serialised data.
3. As a result of the above, you essentially need complete control over the clients that will be reading the data. For example (at least in .net) you can pretty much only properly deserialise with STJ - Newtonsoft can serialise the tokens but doesn't seem capable of deserialising properly.
4. Preserve deserialisation may be a little unreliable - See Should we add support to ignore cycles on serialization? #40099 (comment) (although this is anecdotal).

Advantages

1. Compared to the output of JSON.Net with Ignore, the payload (depending on the complexity and other attributes of the object tree) can be SIGNIFICANTLY SMALLER. This is because with preserved references each isntance is only serialised once, and essentially stored as a variable in the json, then when it's referred to it's just done as a reference to the variable, instead of serialised again like Json.net:

2. It's much faster - STJ utlises new features of .net and types like the Span types and all sorts of other magic of .Net core 3, whereas Newtonsoft is obviously a lot older so only really uses the String types that were available at the time.

The conversation continues over on #40999

This issue was closed a while ago so you're best posting in there.

[ryantgray]

@butler1233 Thanks Lee but your comment did not address my issue. I don't really care about the advantages and disadvantages. I only care about getting it to work. It seems like it should be easy to have someone confirm that you guys implemented changes in JSON 5.0 to address the scenario I'm mentioning and, if so, what needs to be done in a standard default implementation of Blazor WebAssembly that Visual Studio creates to implement it. Also, you comment "this issue was closed a while ago so you're best posting in 40999" but that is closed as well and seems to have nothing to do with my issue.

Can someone please just give me the specific steps including the code changes that need to be made, or point me to them, for what needs to be done in a default implementation of Blazor WebAssembly to set JSON Reference Handling to Preserve? Then, if i have done everything you guys say needs to be done and am still having issues I will take the necessary steps to address them at that point but the first thing i want to do is make sure I have all the necessary changes implemented correctly. Thanks.

[BrunoBlanes]

@ryantgray Again, as mentioned so many times in this thread, ReferenceHandling.Ignore is not supported in S.T.Json, do you wish to implement it using Newtonsoft.Json? I thought you already had it up and working with ReferenceHandling.Preserve.
Keep in mind that we are all doing you a favor, this is not the place to learn to code, questions like yours should be opened at Stackoverflow.

[ryantgray]

@BrunoBlanes It feels like there's a communication disconnect here. Maybe your getting me confused with AlbertoPa? I have never inquired about Ignore. My inquiry was always about Preserve and no, i have never gotten Preserve working. I understand this is not the place to learn code and did start out posting on places like Stackoverflow. However, noone has been able to provide any insight, possibly because you just released your changes to JSON within the last month and the combination of that and Blazor being so new...? If there is a better place/way to address my issue I'd be more than happy to pursue it if i knew what it was. However, I'm trying to implement some functionality released by Microsoft that is not working and so far, noone else seems to have any answers and i'm not even 100% sure the changes you guys made will address the situation in a Blazor WebAssembly application so I feel Microsoft may be the only entity that can address this.

[BrunoBlanes]

It looked to me like the fix for the problem was to update System.Net.Http.Json to version 5.0.0 in the client project and then
implement one of the following two code changes (as described in the referenced articles above) in the ConfigureServices method
of Startup.cs in the Server project:

services.AddControllersWithViews().AddJsonOptions(options =>
{
options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
});

@ryantgray Didn't this fix it for you?

[ryantgray]

That's the way it looked to me to based on the links I provided but no it did not fix it.

[BrunoBlanes]

If you are using Blazor, do also set PropertyNamingPolicy = JsonNamingPolicy.CamelCase or null in your AddJsonOptions method.

[ryantgray]

@BrunoBlanes Thanks, I tried implementing your suggestion with both of the code options I posted and I still get the same errors.

[BrunoBlanes]

Server

services.AddControllersWithViews().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandler = System.Text.Json.Serialization.ReferenceHandler.Preserve;
    options.JsonSerializerOptions.PropertyNamingPolicy = null;
});

Client

var response = await Http.GetFromJsonAsync<T>("{Address}", new JsonSerializerOptions
{
    ReferenceHandler =  System.Text.Json.Serialization.ReferenceHandler.Preserve,
    PropertyNamingPolicy = null
});

Try this and let me know what error are you getting.

[AlbertoPa]

Advantages

1. Compared to the output of JSON.Net with Ignore, the payload (depending on the complexity and other attributes of the object tree) can be SIGNIFICANTLY SMALLER. This is because with preserved references each isntance is only serialised once, and essentially stored as a variable in the json, then when it's referred to it's just done as a reference to the variable, instead of serialised again like Json.net:

I observe a reduction in size in relatively small models or when the EF navigation properties are not fully populated. When pulling complex models, that does not seem to be the case. One example: if I pull an incomplete object with a query, to list objects for CRUD operations in a table, then System.Text.Json gives a smaller payload than Newtonsoft.Json (up to 11 Bytes smaller in my tests over 766 Bytes of objects: 1.4% difference). However, when populating all related data, the situation inverts: System.Text.Json generates a payload 8.27% larger than Newtonsoft. This may be OK if the data size is modest, but it isn't negligible for larger models.

1. It's much faster - STJ utlises new features of .net and types like the Span types and all sorts of other magic of .Net core 3, whereas Newtonsoft is obviously a lot older so only really uses the String types that were available at the time.

This is absolutely the case, hence why it would be good to be able to use it without struggling 😆 . The advantage on the server side is clearly visible when serializing large amounts of data.

Edit: Bruno's latest response works, also with

PropertyNamingPolicy = System.Text.Json.JsonNamingPolicy.CamelCase

at least in my cases. I seem to have some issue when the objects are self-referencing, but I need to troubleshoot more to find what is causing it, because it is not happening systematically 🤷‍♂️

[butler1233]

Ah, my apologies, it looks like I linked to the wrong issue. Should have been #40099. They didnt make the changes to add something like Ignore in 5, although as discussed in that (hopefully correctly linked) issue it's being considered for 6.

…

________________________________ From: Ryan Gray <notifications@github.com> Sent: Sunday, 13 December 2020, 17:01 To: dotnet/runtime Cc: Lee Butler; Mention Subject: Re: [dotnet/runtime] Proposal: Add mechanism to handle circular references when serializing (#30820) @butler1233<https://github.com/butler1233> Thanks Lee but your comment did not address my issue. I don't really care about the advantages and disadvantages. I only care about getting it to work. It seems like it should be easy to have someone confirm that you guys implemented changes in JSON 5.0 to address the scenario I'm mentioning and, if so, what needs to be done in a standard default implementation of Blazor WebAssembly that Visual Studio creates to implement it. Also, you comment "this issue was closed a while ago so you're best posting in 40999" but that is closed as well and seems to have nothing to do with my issue. Can someone please just give me the specific steps including the code changes that need to be made, or point me to them, for what needs to be done in a default implementation of Blazor WebAssembly to set JSON Reference Handling to Preserve? Then, if i have done everything you guys say needs to be done and am still having issues I will take the necessary steps to address them at that point but the first thing i want to do is make sure I have all the necessary changes implemented correctly. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#30820 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADZLJXK4TPBN7VEBZIYGF6DSUTXONANCNFSM4KY4YW5Q>.

[jozkee]

@ryantgray, it seems to me that @BrunoBlanes' suggestion should cover your problem.

Do I also need to update my Server project to change it from targeting .NET Core 3.1 to targeting .NET 5.0?

You need to use at least version 5.0.0 of System.Text.Json in order to be able to use the ReferenceHandler.Preserve feature, this can be achieved by either changing your target .NET version to 5.0 or by installing the latest standalone version of S.T.Json from nuget.

in case you are still having issues, kindly can you please share it with us here and/or in a new issue tagging me. A standalone repro (using the Blazor template for example) would also be very helpful for us.