Deserializing referenced JSON System.Text.Json

[JeremyMahieu]

Description

I've recently switched to using json with references because I had circular objects.

services.AddControllers().AddJsonOptions(options =>
{
   options.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.Preserve;
});

This works fine in one direction, serialization. However when sending back json files with references, the deserialization gives errors.

title: "One or more validation errors occurred.", status: 400
0: "Reference '#/unit' not found. Path: $.unit.bales[0].unit.$ref | LineNumber: 0 | BytePositionInLine: 1204."

This json comes from a Typescript application. I realize there are multiple formats. Presumably I'm using an incorrect format. I know Newtonsoft used to use the $ref $id system and this worked fine. What format does System.Text.Json use?

Configuration

.NET 5.0.100-rc.1.20452.10
Windows 10
Library used to serialize: https://www.npmjs.com/package/json-decycle

[ericstj]

I believe we should handle the same format used by Newtonsoft.

@jozkee does anything stick out about this error?

@JeremyMahieu can you share the payload you're deserializing and the types you're deserializing to?

[jozkee]

@jozkee does anything stick out about this error?

That format looks like what JSON schema uses for backtracking:
https://json-schema.org/understanding-json-schema/structuring.html#using-id-with-ref
Perhaps the user scenario is using both JSON schema and S.T.J ReferenceHandler which might cause conflicts such as this.

I know Newtonsoft used to use the $ref $id system and this worked fine. What format does System.Text.Json use?

We do the same, $id, $ref and $values are the only valid metadata properties.

I wonder, does the same error occurs when you use Newtonsoft?

[jozkee]

Actually, seems like the json-decycle library you linked is using a different format for preserving references:

var result = JSON.stringify(cycled, decycle())
// result === '{"foo":{"bar":{"foo":{"$ref":"#/foo"}}},"bar":{"$ref":"#/foo/bar"}}'

[JeremyMahieu]

Here's a sample program where it works identical to Newtonsoft. ASP.NET also perfect, will try with the original problematic payload tomorrow..
JsonReferencesDemo.zip
JsonReferenceAspNetDemo.zip

[jozkee]

Thanks for the sample; I did a few corrections to your console app given that as is there is no exception happening:

• Added PropertyNameCaseInsensitive to your JsonSerializerOptions given that you are using camel case on your JSON property names.
• Since you are Deserializing into Foo and Foo itself doesn't contain a property named "Foo", I flipped the properties in your JSON: {""bar"":{""foo"":{""$ref"":""#/bar""}}}".

After the changes, I was able to identify what was happening.

When Newtonsoft steps into "$ref":"#/bar", it tries to resolve the reference searching for the id "#/bar", it doesn't find it so null is set and no exception is thrown.

System.Text.Json tries to resolve it as well, but when the reference is unable to resolve, i.e: not found, we throw the JsonException with the error message that you are reporting. This is by design given that the approach of this feature was to be very strict when reading metadata properties ($id, $ref, $values) for security reasons.

You can bypass this validation by implementing your own ReferenceResolver class that doesn't throw on ResolveReference method when the reference isn't found.

If you want me to provide a snippet showing how to do so, let me know, you can also take a look at PreserveReferenceResolver, it is our internal resolver which is used by ReferenceHandler.Preserve.

[JeremyMahieu]

Thank you! So the only reason Newtonsoft works is because it doesn't throw but idealy I want it to work properly so I need to fix my json format.

Any advice on what Javascript/Typescript library I could use that generates the propper $id, $ref and $values format? It seems the "$ref":"#/bar" format is much more common. If no such library exists, why use json this format doesn't have a counterpart in Javascript/Typescript?

This library seems to do it if you set some options JSPON.setSettings({ idFieldName: '$id' });
https://www.npmjs.com/package/jspon
Edit: does not work out of the box the values of id cannot be numbers they must be string. "The '$id' and '$ref' metadata properties must be JSON strings. Current token type is 'Number'. Path: $.$id | LineNumber: 0 | BytePositionInLine: 8.""

[jozkee]

Any advice on what Javascript/Typescript library I could use that generates the propper $id, $ref and $values format?

Unfortunately there are many javascript libraries that deal with this problem on its own unique way; I mentioned flatted and dojo toolkit as an example in the feature specification.

As of now, I couldn't find a javascript counterpart to the Newtonsoft/System.Text.Json format.

[am11]

For Newtonsoft, one recommended library is jsonnetdecycler. The library was quite old (dates back to 2012), and author removed the original bitbucket repository. Luckily, there is a rewrite of that lib in TypeScript available at: https://github.com/Zerq/BridgeBurner/blob/192059fa753aef74bb5e16d4fc9ad0502702c11c/BridgeBurner/BridgeBurner/src/json/jsonnetdecycler.ts.