Developers can customize the JSON serialization contracts of their types

[eiriktsarpalis]

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

   1. It requires that the user owns the type declarations they are looking to customize.
   2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
   3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
   4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.

2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

   1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
   2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects (we're planning on addressing this independently via Developers should be able to pass state to custom converters. #63795).

API Proposal

namespace System.Text.Json 
{
    public sealed partial class JsonSerializerOptions
    {
        public IJsonTypeInfoResolver TypeInfoResolver { [RequiresUnreferencedCode] get; set; }
    }
}

namespace System.Text.Json.Serialization 
{
    public abstract partial class JsonSerializerContext : IJsonTypeInfoResolver
    {
        // Explicit interface implementation calling into the equivalent JsonSerializerContext abstract method
        JsonTypeInfo System.Text.Json.Serialization.Metadata.IJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options);
    }
}

namespace System.Text.Json.Serialization.Metadata 
{
    public interface IJsonTypeInfoResolver
    {
        JsonTypeInfo? GetTypeInfo(Type type, JsonSerializerOptions options);
    }

    // Provides the default reflection-based contract metadata resolution
    public class DefaultJsonTypeInfoResolver : IJsonTypeInfoResolver
    {
        [RequiresUnreferencedCode]
        public DefaultJsonTypeInfoResolver() { }

        public virtual JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options);
        
        public IList<Action<JsonTypeInfo>> Modifiers { get; }
    }

    public static class JsonTypeInfoResolver
    {
        public static IJsonTypeInfoResolver Combine(params IJsonTypeInfoResolver[] resolvers);
    }

    // Determines the kind of contract metadata a given JsonTypeInfo instance is customizing
    public enum JsonTypeInfoKind
    {
        None = 0, // Type is either a primitive value or uses a custom converter -- contract metadata does not apply here.
        Object = 1, // Type is serialized as a POCO with properties
        Enumerable = 2, // Type is serialized as a collection with elements
        Dictionary = 3 // Type is serialized as a dictionary with key/value pair entries
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    // note: added abstract
    public abstract partial class JsonTypeInfo
    {
        public Type Type { get; }
        public JsonSerializerOptions Options { get; }

        // The converter instance associated with the type for the given options instance -- this cannot be changed.
        public JsonConverter Converter { get; }

        // The kind of contract metadata we're customizing
        public JsonTypeInfoKind Kind { get; }

        // Untyped default constructor delegate -- deserialization not supported if set to null.
        public Func<object>? CreateObject { get; set; }

        // List of property metadata for types serialized as POCOs.
        public IList<JsonPropertyInfo> Properties { get; }

        // Equivalent to JsonNumberHandlingAttribute annotations.
        public JsonNumberHandling? NumberHandling { get; set; }

        // Factory methods for JsonTypeInfo
        public static JsonTypeInfo<T> CreateJsonTypeInfo<Τ>(JsonSerializerOptions options) { }
        public static JsonTypeInfo CreateJsonTypeInfo(Type type, JsonSerializerOptions options) { }

        // Factory methods for JsonPropertyInfo
        public JsonPropertyInfo CreateJsonPropertyInfo(Type propertyType, string name) { }
    }

    // remove: [EditorBrowsable(EditorBrowsableState.Never)]
    public abstract partial class JsonTypeInfo<T> : JsonTypeInfo
    {
        // Default typed constructor delegate
        public new Func<T>? CreateObject { get; set; }
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    public abstract partial class JsonPropertyInfo
    {
        public JsonSerializerOptions Options { get; }
        public Type PropertyType { get; }
        public string Name { get; set; }

        // Custom converter override at the property level, equivalent to `JsonConverterAttribute` annotation.
        public JsonConverter? CustomConverter { get; set; }

        // Untyped getter delegate
        public Func<object, object?>? Get { get; set; }

        // Untyped setter delegate
        public Action<object, object?>? Set { get; set; }
    
        // Predicate determining whether a property value should be serialized
        public Func<object, object?, bool>? ShouldSerialize { get; set; }

        // Equivalent to JsonNumberHandlingAttribute overrides.
        public JsonNumberHandling? NumberHandling { get; set; }
    }
}

Usage examples

Custom resolver with constructed JsonTypeInfo

static void Main()
{
    MyType[] values = new MyType[] {
        new() { MyStringId = "123", HideName = true, Name = "John" },
        new() { MyStringId = "124", Name = "James" },
        new() { MyStringId = "-1" },
    };
    JsonSerializerOptions options = new();
    options.TypeInfoResolver = new MyCustomResolver();
    string output = JsonSerializer.Serialize(values, options);
    // [{"ID":123},{"ID":124,"Name":"James"},{}]
}

class MyType
{
    public bool HideName { get; set; }
    public string Name { get; set; }
    public string? MyStringId { get; set; } = "-1";
}

class MyCustomResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        if (type != typeof(MyType))
            return base.GetTypeInfo(type, options);

        JsonTypeInfo<MyType> jti = JsonTypeInfo.CreateJsonTypeInfo<MyType>(options);
        jti.CreateObject = () => new MyType();

        JsonPropertyInfo<int> propId = jti.CreateJsonPropertyInfo<int>("ID");

        propId.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.MyStringId == null ? -1 : ConvertStringIdToInt(m.MyStringId);
        };

        propId.Set = (object o, int val) =>
        {
            MyType m = (MyType)o;
            m.MyStringId = ConvertIntIdToString(val);
        };

        propId.CanSerialize = (object o, int val) =>
        {
            MyType m = (MyType)o;
            return val != -1;
        };

        jti.Properties.Add(propId);

        JsonPropertyInfo<string> propName = jti.CreateJsonPropertyInfo<string>("Name");
        propName.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.Name;
        };

        propName.Set = (object o, string val) =>
        {
            MyType m = (MyType)o;
            m.Name = val;
        };

        propName.CanSerialize = (object o, string val) =>
        {
            MyType m = (MyType)o;
            return !m.HideName && m.Name != null;
        };

        jti.Properties.Add(propName);

        return jti;
    }

    private static int ConvertStringIdToInt(string id) => int.Parse(id);
    private static string ConvertIntIdToString(int id) => id.ToString();
}

Adding support for DataMemberAttribute annotations

public class SystemRuntimeSerializationAttributeResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        JsonTypeInfo jsonTypeInfo = base.GetJsonTypeInfo(type, options);

        if (jsonTypeInfo.Kind == JsonTypeInfoKind.Object &&
            type.GetCustomAttribute<DataContractAttribute>() is not null)
        {
            jsonTypeInfo.Properties.Clear();

            foreach ((PropertyInfo propertyInfo, DataMemberAttribute attr) in type.GetProperties(BindingFlags.Instance | BindingFlags.Public)
                .Select((prop) => (prop, prop.GetCustomAttribute<DataMemberAttribute>() as DataMemberAttribute))
                .Where((x) => x.Item2 != null)
                .OrderBy((x) => x.Item2!.Order))
            {
                JsonPropertyInfo jsonPropertyInfo = jsonTypeInfo.CreateJsonPropertyInfo(propertyInfo.PropertyType, attr.Name ?? propertyInfo.Name);
                jsonPropertyInfo.Get =
                    propertyInfo.CanRead
                    ? propertyInfo.GetValue
                    : null;

                jsonPropertyInfo.Set = propertyInfo.CanWrite
                    ? (obj, value) => propertyInfo.SetValue(obj, value)
                    : null;

                jsonTypeInfo.Properties.Add(jsonPropertyInfo);
            }
        }

        return jsonTypeInfo;
    }
}

Combining resolver

Doc comparing 3 design variants with recommendation can be found here: https://gist.github.com/krwq/c61f33faccc708bfa569b3c8aebb45d6

JsonPropertyInfo vs JsonPropertyInfo<T> vs JsonPropertyInfo<TDeclaringType, TPropertyType>

We have considered different approaches here and it all boils down to perf of the property setter.
According to simple perf tests run on different combinations of declaring types and property types as well 4 different approaches of setters using setter in form of:
delegate void PropertySetter<DeclaringType, PropertyType>(ref DeclaringType obj, PropertyType val);

proves to be overall fastest. Current implementation would require a bit of work for this to be changed and such support can be added later. Given above we've decided to for a time being support only non-generic PropertyInfo with the slowest setter since such type already exists and corresponding setter would have to be added regardless of choice. In the future PropertyInfo<TDeclaringType, TPropertyType> should be added to support for the fastest possible case.

Here are benchmark results: https://gist.github.com/krwq/d9d1bad3d59ff30f8db2a53a27adc755
Here is the benchmark code: https://gist.github.com/krwq/eb06529f0c99614579f84b69720ab46e

Acceptance Criteria

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that should be user customizable in the contract model:

• Object constructors.
• The JSON type of the serialized value: object or array/collection (and possibly primitive values?).
• A list of properties for object types.
• Enumeration/population strategy for array/collection types.
• Serialization callbacks.
• Polymorphic serialization configuration (cf. Developers can use System.Text.Json to serialize type hierarchies securely #63747).
• For the property model:
  • Serialization order.
  • Property name.
  • Getter and setter delegates.
  • Custom converter.
  • Is extension data property.
  • Ignore on serialization/deserialization (cf. [API Proposal]: Add more extensive control over ignoring properties #55781).
  • Number handling.
  • Support modifying (instead of replacing) already initialized properties (cf. Support modifying (rather than replacing) already-initialized properties and fields when deserializing #30258).

Use cases

• JSON/REST transcoding grpc/grpc-dotnet#167
• Support the Specified convention for serializing and deserializing in System.Text.Json #40395
• [API Proposal]: Add more extensive control over ignoring properties #55781
• System.Text.Json: How to serialize only certain properties without using JsonIgnoreAttribute #593
• Add support for MissingMemberHandling to System.Text.Json #37483
• System.Text.Json support to System.Runtime.Serialization #29975
• Each parameter in the deserialization constructor on type 'SpeechCreatedEvent' must bind to an object property or field on deserialization. Each parameter name must match with a property or field on the object. The match can be case-insensitive.' #69547
• Blocking Issue: System.Text.Json and non-public members #31511

Open Questions

• Should we expand the contract model to supporting collection serialization? Is the existing JsonTypeInfo model amenable to supporting collection serialization? Should we consider exposing a JsonCollectionInfo : JsonTypeInfo derived class? Related to Investigate ways to extend deserialization support for arbitrary collection types #38514.

Progress

• Author a working prototype.
• API proposal & review of MVP subset.
• Implementation & tests.
• Add APIs for the remaining items in the Acceptance Criteria.
• Remaining Implementation Work
  • Ensure no size regressions in trimmed applications.
  • System.Text.Json contract customization: not possible to enable deserialization in properties marked JsonIgnoreCondition.Always #71886
  • Reconsider the semantics of JsonProperty.ShouldSerialize null values #71964
  • The JsonSerializerOptions.TypeInfoResolver property should be nullable #71960
  • Add basic regression testing for .NET 6 source generated JsonSerializerContext. #71850
    https://github.com/dotnet/runtime/pull/72044/files#diff-f343f56e41ad406150ed9d35e7548f0d118dd3038c57dff2ff309e5e96331091R588
• Conceptual documentation & blog posts.
  • Improve XML documentation in new APIs.
  • Ensure exception messages are still relevant and up-to-date. Example

cc @steveharter @JamesNK

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

   1. It requires that the user owns the type declarations they are looking to customize.
   2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
   3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
   4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.

2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

   1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
   2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects.

Proposal

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that could be user customizable in the contract model:

• Object constructors.
• The JSON type of the serialized value: object or array/collection (and possibly primitive values?).
• A list of properties for object types.
• Enumeration/population strategy for array/collection types.
• Serialization callbacks.
• Polymorphic serialization configuration (cf. JsonSerializer polymorphic serialization and deserialization support #30083).
• For the property model:
  • Serialization order.
  • Property name.
  • Getter and setter delegates.
  • Custom converter.
  • Is extension data property.
  • Number handling.
  • Support modifying (instead of replacing) already initialized properties (cf. Support modifying (rather than replacing) already-initialized properties and fields when deserializing #30258).

Progress

• Author a working prototype.
• API proposal & review.
• Implementation & tests.
• Conceptual documentation & blog posts.

cc @steveharter @JamesNK

Author:	eiriktsarpalis
Assignees:	eiriktsarpalis
Labels:	area-System.Text.Json, User Story, Priority:0, Cost:L, Team:Libraries
Milestone:	7.0.0

[DRKV333]

I would like to add one more use case for a contract system like this, that would be nice to have support for in System.Text.Json.
Sometimes, especially in dynamically typed languages, people think it's ok to just throw different properties of an object, usually with different types, into a single array.

So, for instance, if you're serializing a person, instead of:

{
    "name": "John",
    "age": 21
}

you'd end up with:
["John", 21]
where the meaning of each item is denoted by their position in the array.

I would like to be able to serialize/deserialize this into a .NET class. Dealing with this currently in System.Text.Json, would involve making a separate converter for each object that is serialized this way, using slow runtime reflection, or coming up with my own fast reflection mechanism (which is a bit redundant, since the serializer already has this internally). In Newtonsoft.Json the situation can be handled by a single generic converter, without any direct use of reflection.

https://github.com/DRKV333/Janki/blob/d3ffc4743b861cb318debbb53ad482ee2a5d9eb6/JankiClientCards/Converters/ObjectToArrayJsonConverter.cs

This setup essentially requires:

• Resolving an object's serialization contract in a converter. JsonSerializer.ContractResolver.ResolveContract
• Enumerating all serializable properties of the object in question, in the right order. JsonObjectContract.Properties
• Creating a new instance of the object. JsonObjectContract.DefaultCreator
• Reading and writing the value of a specific property on an object instance. JsonProperty.ValueProvider.GetValue/SetValue
• Accessing a property's converter to serialize and deserialize the property's value. JsonProperty.Converter

Though if I understand it correctly, most of these should be covered by the current list of planned features.

[Danielku15]

@bartonjs Did you consider the cost of boxing/unboxing in this API design? My concerns mainly are regarding the JsonPropertyInfo members which rely on object at various places. For value types this would mean the serializer has to box/unbox them during serialization.

[eiriktsarpalis]

Did you consider the cost of boxing/unboxing in this API design? My concerns mainly are regarding the JsonPropertyInfo members which rely on object at various places. For value types this would mean the serializer has to box/unbox them during serialization.

Yes. The short answer is boxing structs is necessary in order for setter delegates to work as expected. See #63686 (comment) for more details.

[krwq]

@Danielku15 see the JsonPropertyInfo vs JsonPropertyInfo<T> vs JsonPropertyInfo<TDeclaringType, TPropertyType> section in the issue description. We've found the fastest way is to use ref everywhere but we've also found the current implementation is a bit convoluted to make that work in this release. We've left that option for future since non-generic JsonPropertyInfo will need to exist anyway even if we went with more complex design now. Also we need to make sure that this design will work correctly with polymorphism which is being worked on now as well.

Also note that internally we will use some optimizations to avoid boxing for DefaultJsonTypeInfoResolver so it should have minimal impact but we hope to improve that further and make it public in the future

[krwq]

FYI: main feature branch is already merged and this will be available in preview 6 (or earlier on nightly builds). There are still some improvements planned (see #71123). Please let us know if you have any feedback

[DRKV333]

I'm a little bit confused about how the use-case I mentioned here: #63686 (comment) could be handled with with this new system. I feel like it might be possible, but I couldn't figure it out.
I tried just making a new JsonConverter<>, and using options.TypeInfoResolver.GetTypeInfo(typeToConvert, options) to get the contract for the object I'm currently reading. But this JsonTypeInfo does not have any properties... I'm guessing this is because the type has a custom converter, and in this case the resolver doesn't bother with finding them.
I then tried making a custom IJsonTypeInfoResolver to maybe get access to the original JsonTypeInfo, and then substitute a different one with my custom converter. But it looks like there's no way to supply a different converter from a custom resolver, cause JsonTypeInfo.Converter is read-only...

[krwq]

@DRKV333 unfortunately this scenario is out of scope in this release although we've had long discussion on supporting this - we've decided it's better to support less in this release than supporting everything at once in 2 releases (having said that I'd really wish we could add support for that but at the same time this feature is already large as is and our team has limited resources).

Currently converter determines the JsonTypeInfoKind and also currently there is no way to write custom converter which sets it to value you want, it will always have JsonTypeInfoKind.None which means it won't use properties.
Presumably in the future possible way to support this is by providing abstract converter which can be overriden and has Read/Write which take JsonTypeInfo or just properties and will mean converter knows what to do with properties therefore its kind is Object. There are couple of drawbacks with supporting that in V1: we need to provide way to call serializer recursively. It needs separate API proposal and a bit of thinking how will that work with all existing internal knobs.

FWIW you could possibly write JsonConverterFactory which uses DefaultJsonTypeInfoResolver to get properties for all types you want to serialize as PLIST - if you get into reasonable solution perhaps you will have some ideas how this could be exposed better and create API proposal which we could take into consideration for vNext.

[eiriktsarpalis]

With the bulk of the implementation already merged in Preview 6, we should consider adding the following APIs so that all STJ attribute annotations/interfaces are mapped to the metadata model:

public class JsonTypeInfo
{
    // Maps to IJsonOnSerializing
    public Action<object>? OnSerializing { get; set; }
    // Maps to IJsonOnSerialized
    public Action<object>? OnSerialized { get; set; }
    // Maps to IJsonOnDeserializing
    public Action<object>? OnDeserializing { get; set; }
    // Maps to IJsonOnDeserialized
    public Action<object>? OnDeserialized { get; set; }
}

public class JsonPropertyInfo
{
    // Maps to JsonPropertyOrderAttribute
    public int Order { get; set; } = 0;
    // Maps to JsonExtensionDataAttribute
    public bool IsExtensionData { get; set; }
}

A few remarks on the above:

• We don't expose typed callbacks on JsonTypeInfo<T> since it would have the same issues as typed setters acting on struct types.
• While property order can still be specified implicitly by rearranging elements in the Properties list, this can be cumbersome since there's no way to do in-place sorting in an IList<JsonPropertyInfo> that cannot be set. Users are forced to clear the list and reinsert properties in the desired order. Order would act as a secondary key when sorting properties when the metadata gets locked.
• This only leaves out JsonConstructorAttribute from the model, which is partially addressed by JsonTypeInfo.CreateObject but currently does not provide a way to specify parameterized constructors. Supporting parameterized constructors is nontrivial and should best be addressed in a future release. See Developers can customize the JSON serialization contracts of their types #63686 (comment) for a brief discussion on design considerations.

[eiriktsarpalis]

With .NET 7 work for the feature being mostly done, I have created a user story tracking contract customization work planned for .NET 8: #71967

[krwq]

Since all of the bullet points are now closed and we've shipped what we've planned I think this can be closed now. Any remaining issues should be tracked separately