Extend default value handling in JsonSerializer

[layomia]

Priority scenarios (for .NET 5)

Ignoring default values

See #779 for more info.

We currently have global and per-property options to ignore null when serializing and deserializing reference types. This has peformance benefits, as property getters and setters are called less often. Also, less JSON can be emitted when serializing, meaning that less data can be sent across the wire (if applicable), and there is less data to process when deserializing again.

We should add options to ignore default values for value types.

Passing null to custom converters

API approved in #34439. Click for more info.

Null values are not passed to converters for reference types. Users may wish to handle (de)serialization of types and properties with custom converters. This should include null values and JSON tokens, if so desired.

For example, in the Cosmos v3 database, when deserializing a CRS class (Coordinate Reference System) instance and the input JSON is null, a new UnspecifiedCrs instance is set instead.

A workaround here is to set the default value during instantiation of the type, and ignore null values on deserialization. This may have a noticeable perf impact if multiple properties are set to custom defaults during object instantiation, only for several of them to be set again when deserializing.

API proposal

See updated proposal following review feedback - #35649 (comment).

Initial proposal (click to view)

Option to ignore default values globally

Add IgnoreDefaultValues default values option that can be applied to both reference and value types.

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether null values are ignored during serialization and deserialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        [Obsolete("This property is obsolete. Use IgnoreDefaultValues instead.", error: false)]
        public bool IgnoreNullValues { get; set; }

        /// <summary>
        /// Determines whether default values are ignored during serialization and deserialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool IgnoreDefaultValues { get; set; }
    }
}

It would be invalid for a caller to set both IgnoreNullValues & IgnoreDefaultValues.

What about people who want to ignore null values, but include value-type default values?

They can use [JsonIgnore(Condition = Never)] on each property to include. See #35649 (comment) for an alternate approach that has more granularity.

Option to ignore default values per property

Rename JsonIgnoreCondition.WhenNull to JsonIgnoreCondition.WhenDefault.
This is a breaking change between preview 4 & 5.

namespace System.Text.Json
{
    /// <summary>
    /// Controls how the <see cref="JsonIgnoreAttribute"/> ignores properties on serialization and deserialization.
    /// </summary>
    public enum JsonIgnoreCondition
    {
        /// <summary>
        /// Property will always be ignored.
        /// </summary>
        Always = 0,
        /// <summary>
        /// Property will only be ignored if it is the default value.
        /// </summary>
        WhenDefault = 1,
        /// <summary>
        /// Property will always be serialized and deserialized, regardless of <see cref="JsonSerializerOptions.IgnoreNullValues"/> configuration.
        /// </summary>
        Never = 2
    }
}

API usage

Ignoring default values

public class MyClass
{
    public int MyInt { get; set; }
    public bool MyBool { get; set; }
    public string MyString { get; set; }
}

public class MyClassWithInitializedMembers
{
    public int MyInt { get; set; } = 123;
    public bool MyBool { get; set; } = true;
    public string MyString { get; set; } = "abc";
}

var options = new JsonSerializerOptions
{
    IgnoreCondition = JsonIgnoreCondition.WhenWritingDefaultValues
};

var obj = new MyClass();

// Serialize (defaults ignored)
string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {}

// Deserialize (defaults not ignored)
json = @"{""MyInt"":0,""MyBool"":false,""MyString"":null}";
var obj2 = JsonSerializer.Deserialize<MyClassWithInitializedMembers>(json, options);
Console.WriteLine(obj2.MyInt); // 0
Console.WriteLine(obj2.MyBool); // false
Console.WriteLine(obj2.MyString); // null

Ignoring default values globally but including some properties:

public class MyClass
{
    public int MyInt { get; set; }
    [JsonIgnore(Condition = JsonIgnoreCondition.Never)]
    public bool MyBool { get; set; }
    public string MyString { get; set; }
}

var options = new JsonSerializerOptions
{
    IgnoreCondition = JsonIgnoreCondition.WhenWritingDefaultValues
};

var obj = new MyClass();

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"MyBool":false}

Hand-picking properties to ignore on serialization when default

public class MyClass
{
    public int MyInt { get; set; }
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefaultValues)]
    public bool MyBool { get; set; }
    public string MyString { get; set; }
}

var obj = new MyClass();

string json = JsonSerializer.Serialize(obj);
Console.WriteLine(json); // {"MyInt":0,"MyString":null}";

Additional features (5.0 backlog/future)

Ability to specify a custom default value

Users may want to specify a custom default value to be used instead of the type default.

Replacing type defaults or missing values with custom defaults when serializing and deserializing

EDIT: After discussing with @JamesNK, it was discovered that this feature, which is equivalent to applying DefaultValueHandling.Populate was not created in Newtonsoft.Json to solve any particular customer scenario, and is likely not useful for many customers. Thus, API for this scenario will not be prioritized.

This feature allows a custom default value to be used instead of the property's actual value:

• when the property value to serialize or deserialize is the default value for the type
• when there's no value for a property/field in the JSON payload when deserializing

---

Altogether, these features give JsonSerializer parity with Newtonsoft.Json's serializer around default value handling, excluding programatic configuration capabilities enabled with exposing JsonClassInfo, JsonPropertyInfo & friends; resulting in a model similar to using a custom [contract resolver(https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_Serialization_DefaultContractResolver.htm). In this model, we can expose a Predicate<T> based model for determining whether to ignore or replace the value of a property, similar to Newtonsoft's ShouldSerialize and ShouldDeserialize properties.

API proposals for these features (click to expand).

API proposals

Option for users to specify a custom default value for a type, property, or field

The API is designed such that reference-type properties do not have to share the same instance as the default value. The default value provided overrides the type default.

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Specifies the default value for a type, property, or field.
    /// </summary>
    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Property | AttributeTargets.Field, AllowMultiple = false)]
    public class DefaultValueProviderAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="DefaultValueProviderAttribute"/> with the specified provider type.
        /// </summary>
        /// <param name="providerType">The type of the converter.</param>
        public DefaultValueProviderAttribute(Type providerType)
        {
            ProviderType = providerType;
        }

        /// <summary>
        /// Initializes a new instance of <see cref="DefaultValueProviderAttribute"/> with the specified default value.
        /// </summary>
        /// <param name="defaultValue">The default value.</param>
        /// <exception cref="ArgumentNullException">
        /// Thrown if <paramref cref="defaultValue" /> is <see langword="null" />.
        /// </exception>
        public DefaultValueProviderAttribute(object defaultValue)
        {
            DefaultValue = defaultValue;
        }

        /// <summary>
        /// Initializes a new instance of <see cref="DefaultValueProviderAttribute"/>.
        /// </summary>
        protected DefaultValueProviderAttribute() { }

        /// <summary>
        /// The type of the provider to create. If null and DefaultValue is null, <see cref="CreateProvider"/> will be used to obtain the provider.
        /// </summary>
        public Type? ProviderType { get; private set; }

        /// <summary>
        /// The default value for the type, if specified during construction. If null and provider type is null, <see cref="CreateProvider"/> will be used to create the provider.
        /// </summary>
        public object DefaultValue { get; private set; }

        /// <summary>
        /// If overridden and both <see cref="ProviderType"/> and <see cref="DefaultValue"/> are null, allows a custom attribute
        /// to create the default value provider in order to pass additional state.
        /// </summary>
        /// <returns>
        /// The custom default value provider.
        /// </returns>
        public virtual DefaultValueProvider? CreateProvider(Type typeToConvert)
        {
            return null;
        }
    }

    /// <summary>
    /// Provides a default value for an object or value when converting to or from JSON.
    /// </summary>
    public abstract partial class DefaultValueProvider { }

    /// <summary>
    /// Provides a default value for an object or value when converting to or from JSON.
    /// </summary>
    /// <typeparam name="T">The <see cref="Type"/> to convert.</typeparam>
    public abstract partial class DefaultValueProvider<T> : DefaultValueProvider
    {
        /// <summary>
        /// Provides a default value for the type.
        /// </summary>
        public abstract T GetDefaultValue();
    }
}

Notes

• The DefaultValueProviderAttribute is provided to accomodate callers who
  • want to specify defaults for simple/literal types
  • don't care about shared instances for reference types
  • don't care about the perf hit of unboxing for value types every time we set the default value
• The specified T must be the same as any custom converter specified for the type; otherwise, typeof(T) must be equal to the type.
• The non-generic DefaultValueProvider base class is useful to provide future functionality
  • e.g. adding an abstract bool IsCompatibleWith(Type) method that will be used if we add the ability to specify the default value for a type at runtime, similar to JsonConverter.CanConvert
  • If we don't add this DefaultValueProvider type, then the return type of CreateProvider above will have to be typeof(object?).

Optional addition: extend JsonConverter<T>

namespace System.Text.Json.Serialization
{
    public abstract partial class JsonConverter<T> : JsonConverter
    {
        /// <summary>
        /// Indicates if the converter can provide a default value for <typeparamref name="T">.
        /// </summary>
        public virtual bool CanProvideDefaultValue => false;
        
        /// <summary>
        /// Provides a default value for the type.
        /// </summary>
        public abstract T GetDefaultValue();
    }
}

If a custom converter is provided for the type, it can opt into providing a default value for the type. This avoids having to specify a converter and a default value provider.

Option to globally specify when to replace values with a custom default

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Specifies the condition that must be met before a property will be populated with a custom default value.
        /// The default value is <see cref="JsonPopulateCondition.Never"/>
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public JsonPopulateCondition PopulateCondition { get; set; } = JsonPopulateCondition.Never;
    }
}

It is invalid for a custom default value not to be specified when PopulateCondition is set to a value other than JsonPopulateCondition.Never.

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Indicates that the property should be set to the custom default value if the specified condition is met.
    /// </summary>
    [AttributeUsage(AttributeTargets.Property, AllowMultiple = false)]
    public sealed class JsonPopulateAttribute : JsonAttribute
    {
        /// <summary>
        /// Specifies the condition that must be met for a property to be set with a custom default value/>.
        /// </summary>
        public JsonPopulateCondition Condition { get; }

        /// <summary>
        /// Initializes a new instance of <see cref="JsonPopulateAttribute"/>.
        /// </summary>
        public JsonPopulateAttribute(JsonPopulateCondition condition) => Condition = condition;
    }
}

It is invalid for a custom default value not to be specified when this attribute is used.

Option to specify a condition for a property or field to be set to a custom default

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Controls how the <cref="JsonPopulateAttribute" /> will decide to populate properties or fields with a custom default value.
    /// </summary>
    [Flags]
    public enum JsonPopulateCondition
    {
        /// <summary>
        /// The operation should never be carried out on the property or field.
        /// </summary>
        Never,
        /// <summary>
        /// The operation should only be carried out if the value of the property or field is the CLR default value for the type.
        /// </summary>
        WhenTypeDefault,
        /// <summary>
        /// The operation should only be carried out if there's no value for the property in the input JSON payload when deserializing.
        /// </summary>
        WhenMissing,
        /// <summary>
        /// The operation should be carried out if the value of the property or field is the CLR default value when serializing or deserializing,
        /// or if there's no value for the property in the input JSON payload when deserializing.
        /// </summary>
        WhenTypeDefaultOrMissing
    }
}

Interaction with converters handling null on serialization and deserialization

The behavior when null is encountered when (de)serializing a property is as follows:

• If HandleNull is true:

  • If JsonPopulate is active on the property:

    • the converter's Read/Write method will not be called, and the specified default will be set.

  • If JsonPopulate is not active:

    • the converter's Read/Write method will be called with the null value as an input.

• If HandleNull is false:

  • If JsonPopulate is active on the property:

    • the specified default will be set

  • If JsonPopulate is not active on the property:

    • the serializer will handle the (de)serialization as done today

If IgnoreDefaultValues is active, then null is not passed to the converter. This can be overriden on a property or field using JsonIgnoreAttribute with the Condition set to Never.

Using the ShouldSerializeXxx pattern

Using the ShouldSerializeXxx pattern to indicate whether a property should be serialized (in the context of ignoring default values) was considered but not pursed for the following reasons:

• Extra start-up cost to fetch the corresponding class method for each property and create a performant delegate to it.
• Works only for serialization
  • would have to introduce ShouldDeserializeXxx as well
• This approach is not utile for the "replace" functionality.
• This approach can be added in the future if we have an equivalent to custom contract resolvers (using the Should(De)Serialize models mentioned above).

Examples

The "replace" operation occurs before the "ignore" operation.

Given a type MyClass and and a default value provider for the string type:

public class MyClass
{
    [DefaultValueProvider(typeof(StringDefaultValueProvider))]
    [JsonPopulate(JsonPopulateCondition.WhenTypeDefault)]
    public string MyString { get; set; }
}

public class StringDefaultValueProvider : DefaultValueProvider<string>
{
    public override string GetDefaultValue() => "Hello, world!";
}

var options = new JsonSerializerOptions
{
    IgnoreNullValues = true
};

string json = @"{""MyString"":null}";
MyClass obj = JsonSerializer.Deserialize<MyClass>(json, options);
Console.WriteLine(obj.MyString); // Hello, world!

obj.MyString = null;
Console.WriteLine(JsonSerializer.Serialize(obj, options)); // {"MyString":"Hello, world!"}

Populating with a custom default when there's no JSON value for the property, or the property is the CLR default.

Given a MyClass class:

public class MyClass
{
    [DefaultValueProvider(typeof(StringDefaultValueProvider))]
    [JsonPopulate(Condition = JsonPopulateCondition.WhenMissing)]
    public string MyString { get; set; }
}

string json = @"{}";
MyClass obj = JsonSerializer.Deserialize<MyClass>(json);
Console.WriteLine(obj.MyString); // Hello, world!;

obj.MyString = null;
Console.WriteLine(JsonSerializer.Serialize(obj)); // {"MyString":"Hello, world!"}

[ghost]

Tagging subscribers to this area: @jozkee
Notify danmosemsft if you want to be subscribed.

[scalablecory]

In the past, I've wanted to ignore null but not default. What do you think about making this an enum instead? Something like.

enum JsonDefaultValuePolicy
{
   SerializeAll,
   IgnoreNull,
   IgnoreDefault,
   IgnoreNullOrDefault
}

The property comment says default will be ignored on deserialization too. Am I understanding the behavior correctly here?:

class MyClass
{
   public int Foo { get; set; } = 123;
}

MyClass c = Deserialize<MyClass>("{'Foo': 0}");
Assert.Equal(123, c.Foo);

[layomia]

In the past, I've wanted to ignore null but not default. What do you think about making this an enum instead?

That's a legit scenario. Rather than add a new enum, we could leverage the existing JsonIgnoreCondition enum:

enum JsonIgnoreCondition
{
    Always (per-property default when used as `[JsonIgnore] condition`; invalid as global option),
    WhenNull,
+   WhenValueDefault,
+   WhenDefault,
    Never (global default)
}

Alternatively, instead of making IgnoreNullValues obsolete, we could have IgnoreNullValues apply to reference types (as done today), and IgnoreDefaultValues would apply to both reference and value types. This would require clear documentation.

FWIW, with the current API proposal you could prevent value-type defaults from being ignored by applying [JsonIgnore(Condition = JsonIgnoreCondition.Never)] on each value-type property. This may be inconvenient if there are lots of properties to annotate.

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether null values are ignored during serialization and deserialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        [Obsolete("This property is obsolete. Use IgnoreDefaultValues instead.", error: false)]
        public bool IgnoreNullValues { get; set; }

        /// <summary>
        /// Determines when property values are ignored during serialization and deserialization.
        /// The default value is <see cref="JsonIgnoreCondition.Never" />.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public JsonIgnoreCondition IgnoreCondition { get; set; } = JsonIgnoreCondition.Never;
    }
}

---

The property comment says default will be ignored on deserialization too. Am I understanding the behavior correctly here?:

Yes, that's the behavior I'm proposing. Wanting to ignore value-type defaults on deserialization is probably an edge case, but it may have some perf benefits if a bunch of 0s and falses are to be ignored as we don't have to call the property setter. However, the cost of parsing the JSON may eclipse the cost of calling the setter in some cases. I'll write a benchmark to see.

FWI Newtonsoft ignores default values on deserialization when the option is active. Differing here might come as a surprise to some users:

using System;
using Newtonsoft.Json;
					
public class Program
{
	public static void Main()
	{
		var settings =  new JsonSerializerSettings
		{
			DefaultValueHandling = DefaultValueHandling.Ignore,
		};
		
		string json = @"{""MyInt"": 0}";
		var obj = JsonConvert.DeserializeObject<MyClass>(json, settings);
		Console.WriteLine(obj.MyInt); // 123
	}
	
	public class MyClass
	{
		public int MyInt { get; set; } = 123;
	}
}

[bartonjs]

• IgnoreDefaultValues doesn't seem like it makes sense on Deseralization. Rename to SerializeDefaultValues, which defaults to true.
• Obsoleting IgnoreNullValues may not make sense, there's an outstanding question on this.
  • If it is being obsoleted, it should be marked [EditorBrowsable(EditorBrowsableState.Never)] in this version.
• Changing JsonIgnoreCondition.WhenNull is being deferred to some usage questions on deserialization.

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        [Obsolete("This property is obsolete. Use IgnoreDefaultValues instead.", error: false)]
        [EditorBrowsable(EditorBrowsableState.Never)]
        public bool IgnoreNullValues { get; set; }

        public bool SerializeDefaultValues { get; set; } = true;
    }
}

[layomia]

Following feedback, @steveharter and I spoke with @JamesNK about whether providing an option to ignore default values when deserializing should be prioritized. James has not received much feedback about this, and doesn't think that it is a major scenario.

Here's the updated proposal:

Modify option to specify when to ignore properties

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Controls how the <see cref="JsonIgnoreAttribute"/> ignores properties on serialization and deserialization.
    /// </summary>
    public enum JsonIgnoreCondition
    {
        /// <summary>
        /// Property will always be serialized and deserialized.
        /// </summary>
        Never = 0,
        /// <summary>
        /// Property will always be ignored when serializing and deserializing.
        /// </summary>
        Always = 1,
        /// <summary>
        /// Property will be ignored when serializing if it is the default value.
        /// </summary>
        WhenWritingDefaultValues = 2
    }
}

This enum was added in .NET 5 preview 3.

In the future, we could add WhenReadingDefaultValues and WhenDefaultValue (for both serialization and deserialization).

The current values are Always, WhenNull (to be removed), and Never.

Today, JsonIgnoreAttribute has a Condition property whose default value is JsonIgnoreCondition.Always.

Provide global option to ignore default values when serializing

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Specifies a condition to determine when properties with default values are ignored during serialization or deserialization.
        /// The default value is <see cref="JsonIgnoreCondition.Never" />.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred,
        /// or if this property is set to <see cref="JsonIgnoreCondition.Always"/>
        /// </exception>
        public JsonIgnoreCondition DefaultIgnoreCondition { get; set; } = JsonIgnoreCondition.Never;
    }
}

This approach gives us flexibility to allow ignoring default values when deserializing in the future (by adding a new enum value to JsonIgnoreCondition).

Alternate approach (click to view)

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether default values are ignored during serialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool IgnoreDefaultValues { get; set; }
    }
}

We prefer for this to be named in a way that the default value is false. This is to be consistent with the other options on JsonSerializerOptions.

If desired in the future, more properties would need to be added to JsonSerializerOptions to allow ignoring default values when deserializing.

Make IgnoreNullValues obsolete

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether null values are ignored during serialization and deserialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        [Obsolete("This property is obsolete. Use DefaultIgnoreCondition instead.", error: false)]
        [EditorBrowsable(EditorBrowsableState.Never)]
        public bool IgnoreNullValues { get; set; }
    }
}

[terrajobst]

Video

• Looks great. Only suggestion is to change WhenWritingDefaultValues to WhenWritingDefault.
• JsonSerializerOptions.DefaultIgnoreCondition should throw ArgumentException rather than InvalidOperationException

namespace System.Text.Json.Serialization
{
    public enum JsonIgnoreCondition
    {
        Never = 0,
        Always = 1,
        WhenWritingDefault = 2
    }    
}

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public JsonIgnoreCondition DefaultIgnoreCondition { get; set; } = JsonIgnoreCondition.Never;
    }
}

[layomia]

Closing this composite issue as there are tracking issues for the sub-tasks:

• Converters handling null: Provide opt-in for custom converters to handle null #34439
• Options to ignore default values during serialization: System.Text.Json option to ignore default values during serialization #779.
• Option to provide a custom default value: [JsonSerializer] Provide option to specify a custom default value for a type, property, or field #36236