System.Text.Json: don't emit null values when serializing collections

[SanderSade]

System.Text.Json has JsonSerializerOptions.IgnoreNullValues, but this only applies to properties, and will not ignore values in collections.

For example:

var objects = new Dictionary<string, object> 
    { { "a", "!" }, { "b", 1 }, { "c", null }, { "d", new object() }, { "e", DBNull.Value } };
var json = JsonSerializer.Serialize(objects, 
    new JsonSerializerOptions { IgnoreNullValues = true, WriteIndented = true });
Console.WriteLine(json);

Output is:

{
  "a": "!",
  "b": 1,
  "c": null,
  "d": {},
  "e": {}
}

(DbNull.Value has #418 open already).

Expectation for IgnoreNullValues = true is that all null values would be ignored and not serialized, not just properties. However, this would be a breaking change from JSON.NET, which has the same behavior with NullValueHandling.Ignore, so probably a new JsonSerializerOptions property is needed (no idea what it could be named).

This is especially useful in a very common scenario (1, 2, 3 etc) when data is fetched from database with SqlDataReader as List<Dictionary<string, object>>.

[ericstj]

@layomia @ahsonkhan

[ahsonkhan]

However, this would be a breaking change from JSON.NET, which has the same behavior with NullValueHandling.Ignore,

Yep. When using Newtonsoft.Json, how did you workaround this issue given it behaves similarly for null value handling within collections?

From @steveharter

Having null values in collections\arrays is by design and is necessary for correctness since collections\arrays likely have null values for a reason. From past discussions ignoring null values was primarily for perf (smaller JSON on serialization and less work to do on deserialize if there are nulls in the JSON) and in the case of deserialization may help prevent "default" values from being overwritten with null, although we really need per-property support for that instead of just a global for that argument.

So, this begs the questions, what is your scenario in which you need to ignore null values in collections and do you need per-property/collection granularity to turn this feature on, or is a global switch what you need? Also, why is ignoring nulls in collections, and hence changing length/count as a side effect, not a concern for your use case?

[SanderSade]

Yep. When using Newtonsoft.Json, how did you workaround this issue given it behaves similarly for null value handling within collections?

There are several options. Don't add null-valued entries to a dictionary, or filter them out after adding, as well as using custom converters or even ShouldSerialize... methods. A cursory search in Stack Overflow finds many posts - so if we have an option to improve what seems to be a common scenario/question, we could take it.

So, this begs the questions, what is your scenario in which you need to ignore null values in collections and do you need per-property/collection granularity to turn this feature on, or is a global switch what you need? Also, why is ignoring nulls in collections, and hence changing length/count as a side effect, not a concern for your use case?

We have at the moment multiple scenarios where result.Add(Enumerable.Range(0, reader.FieldCount) .ToDictionary(reader.GetName, reader.GetValue)); pattern is used (for the record, I am not a fan of it. Quite the opposite.). Many of the values can be null, often comprising the majority of the returned JSON. Excluding null values from JSON would reduce the overhead for both serialization and network use. This, of course, could be also achieved with value-based filtering, same as above, with similar O(n)+ overhead.

On granularity, a JsonSerializerOptions property would probably be preferable, allowing the developer to decide if a global options are followed or apply the behavior just to specific cases.

[layomia]

From @JamesNK

Properties can default to null, so excluding null properties from serialized JSON is fine. Minewhile, position in a collection can be important.

Even given all the correctness and round-tripping considerations, one may desire to exclude null colleciton elements when serializing to reduce JSON size or for other reasons, and not pay the cost of pre-filtering.

If a global and/or per-property option is added, should it apply to deserialization as well, i.e. don't add null elements to my collections?

[ahsonkhan]

From @CodeBlanch in #31786

Porting a service from .NET Framework to .NET Core we noticed a lot of NULLs started showing up in Kibana for certain fields (but not all fields) pushed from our structured json logs. I tracked this down to a bug in the handling of JsonExtensionData with IgnoreNullValues=true.

Consider this class:

public class TestClass
{
   public string Prop1 { get; set; } = "value";
   public string Prop2 { get; set; }
}

With IgnoreNullValues=true that will serialize as:

{"Prop1":"value"}

Now consider this class:

public class TestClass
{
   [JsonExtensionData]
   public IDictionary<string, object> Fields { get; set; } = new Dictionary<string, object>
   {
      ["Prop1"] = "value",
      ["Prop2"] = null
   }
}

Should be identical with IgnoreNullValues=true, but ends up as:

{"Prop1":"value","Prop2":null}

[ahsonkhan]

The IgnoreNullValues flag doesn't apply to collections (for example, list, dictionary, etc.) and only applies to when property values themselves is null. This includes dictionaries that are annotated with [JsonExtensionData]. This is by-design and matches Newtonsoft.Json behavior

Changing this would be a breaking change.

For example:

public static void Main()
{
    var options = new JsonSerializerOptions { IgnoreNullValues = true };
    var obj = new TestClass();

    // {"List":["a",null,"b"]}
    Console.WriteLine(JsonSerializer.Serialize(obj, options));

    // {"List":["a",null,"b"]}
    Console.WriteLine(JsonConvert.SerializeObject(obj, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }));
}

public class TestClass
{
    public List<string> List { get; set; } = new List<string> { "a", null, "b" };
}

See: #29495 (comment)

[eiriktsarpalis]

Closing per feedback.