[JsonSerializer] Relax restrictions on ctor param type to immutable property type matching where reasonable

[JSkimming]

Description

I'm trying to deserialise an object from a json string. In this case the constructor allows a nullable value type as the parameter, setting a non-null property (defaulting if null).

I expect (since it is the behaviour with Newtonsoft.Json) the serializer to handle compatible constructor and bound value type properties (or fields) where the only difference is one is nullable.

The following demonstrates the issue:

public class Example
{
    public Example(Guid? aGuid) => AGuid = aGuid ?? Guid.Empty;
    public Guid AGuid { get; }
}

Example original = new Example(Guid.NewGuid());

// Works with Newtonsoft.Json.
string withJsonNet = JsonConvert.SerializeObject(original);
JsonConvert.DeserializeObject<Example>(withJsonNet);

// Fails with System.Text.Json.
string withSystemTextJson = System.Text.Json.JsonSerializer.Serialize(original);
System.Text.Json.JsonSerializer.Deserialize<Example>(withSystemTextJson);

An InvalidOperationException is thrown from the method System.Text.Json.JsonSerializer.Deserialize :

System.InvalidOperationException: Each parameter in constructor 'Void .ctor(System.Nullable`1[System.Guid])' on type 'Example' must bind to an object property or field on deserialization. Each parameter name must match with a property or field on the object. The match can be case-insensitive.

Configuration

I'm building an ASP.NET Core app targeting netcoreapp3.1, and using version 5.0.0-rc.2.20475.5 of System.Text.Json. I'm also using version 12.0.3 of Newtonsoft.Json.

Other information

Stack Trace:

System.InvalidOperationException: Each parameter in constructor 'Void .ctor(System.Nullable`1[System.Guid])' on type 'Example' must bind to an object property or field on deserialization. Each parameter name must match with a property or field on the object. The match can be case-insensitive.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ConstructorParameterIncompleteBinding(ConstructorInfo constructorInfo, Type parentType)
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](Utf8JsonReader& reader, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at <custom code>

[layomia]

Why make the constructor parameter nullable? The AGuid property is non-nullable which means that null will never be serialized. Are you expecting null JSON tokens to bind with the parameter on deserialization? Where would this JSON come from?

[JSkimming]

Thanks for the reply @layomia

To present a minimal use-case the example is intentionally trivial.

My situation is rather more complex. We have several scenarios where having the parameter as nullable was desirable (and worked well using Newtonsoft.Json), some of which we can, and have, resolved by making the parameter non-nullable.

But there are other scenarios where a nullable parameter with a non-nullable property is still preferred. One being where we have an unknown number of legacy objects that are serialised to a store with null values. The use-cases have since been updated where new instances are not serialised with null, but we still want to support the de-serialization of the legacy objects.

If we had been using System.Text.Json at the time, we probably would have implemented the solution differently, but as I highlighted above Newtonsoft.Json worked.

Hope the context helps.

Ultimately though, this is a difference in behaviour with Newtonsoft.Json. It looks like you are trying to resolve edge cases in to make System.Text.Json "the standard JSON stack for .NET." (See Issue #43620). And this is one such edge case.

[layomia]

From @https://github.com/NN--- in #46480:

Description

    public class Q
    {
        [JsonConstructor]
        public Q(int? x)
        {
            if (x is null) throw new Exception();
            X = x.Value;
        }
        
        public int X { get; }
    }

Given string:

var str = JsonSerializer.Deserialize<Q>("{\"X\":123}" ");

It is deserialized fine with Newtonsoft, but not with System.Text.
The reason is that constructor parameter type is int? while the property has type of int.
Newtonsoft doesn't validate it, giving an option to do anything in the constructor.

There are possible workarounds but they are not as simple as the original code:

Make property private and add a public non nullable:

    public class Q
    {
        [JsonConstructor]
        public Q(int? x)
        {
            if (x is null) throw new Exception();
            X = x.Value;
        }

        public int? X { get; }

        [JsonIgnore]
        public int NonNullableX => X!.Value;
    }

Annotate nullable as non nullable, however it requires explicit call to Value.:

    public class Q
    {
        [JsonConstructor]
        public Q(int? x)
        {
            if (x is null) throw new Exception();
            X = x.Value;
        }

        [DisallowNull][NotNull]
        public int? X { get; }
    }

Configuration

Regression?

Not a regression.

Other information

[layomia]

We could look into relaxing the matching algorithm here if it proves to be unwieldy. The workaround here is to refactor the POCO code slightly to make the property type and the constructor parameter type the same. This issue will not be treated as high priority until a blocked and non-trivial scenario is provided.

[GabeDeBacker]

Deserializing JSON into an object that can be bound to WPF\XAML is likely very common place and converting an incoming IEnumerable into a observable collection that XAML can use is also likely common.

Not supporting this limits System.Text.Json's use with an XAML\WPF\UWP apps.

[JSkimming]

This issue will not be treated as high priority until a blocked and non-trivial scenario is provided.

@layomia As I explained, the example is trivial to present a minimal use-case. I also went on to explain the non-trivial scenario:

we have an unknown number of legacy objects that are serialised to a store with null values. The use-cases have since been updated where new instances are not serialised with null, but we still want to support the de-serialization of the legacy objects.

We are also blocked. Our workaround is to stick with Newtonsoft.Json.

[layomia]

Thanks for the responses and expanding on the importance here. There are definitely various scenarios where loosening the matching restrictions is helpful. We can address this in the .NET 6.0 release.

[layomia]

From @GabeDeBacker in #47422:

Description

System.Text.Json deserialization requires that a property type match the constructor type for immutable properties even though the constructor can convert the type.

This is a simple example of a class that will convert an incoming IEnumerable to a ReadOnlyObservableCollection for XAML binding.

        [JsonConstructor]
        public Logger(IEnumerable<LogEntry> entries)
        {
            this.Entries = new ReadOnlyObservableCollection<LogEntry>(this.entries);
        }

        public ReadOnlyObservableCollection<LogEntry> Entries { get; }

When desrializing from JSON, this fails.

Changing the property to be IEnumerable allows the deserialization to succeed, but that means I would need to add “another” property to this class for XAML binding to work. (Which is what this class is used for). The below just doesn’t seem right and was not something I had to do when using NewtonSoft

        public Logger(IEnumerable<LogEntry> entries)
        {
            this.Entries = entries;
            this.ObersvableEntries = new ReadOnlyObservableCollection<LogEntry>(this.entries);
        }

        public IEnumerable<LogEntry> Entries { get; }

        [JsonIgnore]
        public ReadOnlyObservableCollection<LogEntry> ObersvableEntries { get; }

[layomia]

Today, we expect an exact match between the constructor parameter type and the immutable property type. This is too restrictive in two major cases:

• mapping nullable value-type ctor args to immutable properties of a non-nullable version of the type, as described in the description above [JsonSerializer] Relax restrictions on ctor param type to immutable property type matching where reasonable #44428 (comment):

  public class Example
  {
      public Guid AGuid { get; }
      public Example(Guid? aGuid) => AGuid = aGuid ?? Guid.Empty;
  }

• scenarios where a reasonable conversion can be made between the ctor param type and the destination immutable property type, for example the scenario in [JsonSerializer] Relax restrictions on ctor param type to immutable property type matching where reasonable #44428 (comment):

  public class Logger
  {
      public ReadOnlyObservableCollection<LogEntry> Entries { get; }
      public Logger(IEnumerable<LogEntry> entries) => Entries = new ReadOnlyObservableCollection<LogEntry>(entries);
  }

We can loosen the restriction and support these scenarios by checking that the ctor parameter is assignable from the immutable property type. This new algorithm is in accordance with the serializer always round-tripping (i.e being able to deserialize whatever we serialize), and maintains a high probability that the incoming JSON is compatible with the target ctor param. This is important to avoid unintentional data loss.

If there are more reasonable scenarios that will not be satisfied with this proposal, we can evaluate them and perhaps adjust further.

[layomia]

We can also, or alternatively, consider a property on JsonConstructorAttribute, that indicates no restriction between the ctor parameter type and the immutable property type. This would allow them to be two arbitrarily different types, basically an "I know what I'm doing mode". It would still be required for their CLR names to match.

/// <summary>
/// When placed on a constructor, indicates that the constructor should be used to create
/// instances of the type on deserialization.
/// </summary>
[AttributeUsage(AttributeTargets.Constructor, AllowMultiple = false)]
public sealed class JsonConstructorAttribute : JsonAttribute
{
    /// <summary>
    /// When <see cref="true" />, indicates that no restriction should be placed on the types of a constructor
    /// parameter and a property when there is a case-insensitive match between their names.
    /// </summary>
    public bool UseRelaxedPropertyMatching { get; set; }

    /// <summary>
    /// Initializes a new instance of <see cref="JsonConstructorAttribute"/>.
    /// </summary>
    public JsonConstructorAttribute() { }
}

A global option can be considered as well, to support non-owned types where we can't decorate with an attribute:

public sealed class JsonSerializerOptions
{
    /// <summary>
    /// When <see cref="true" />, indicates that no restriction should be placed on the types of a constructor
    /// parameter and a property when there is a case-insensitive match between their names.
    /// </summary>
    public bool ConstructorUseRelaxedPropertyMatching { get; set; }
}

---

All of this design assumes that there will always be a requirement that every constructor parameter binds to an object property, per the original spec for this feature: #33095.

[GabeDeBacker]

@layomia - Implementing either (or both) of what you mentioned above (the JsonConstructorAttribute argument or is the constructor argument assignable from the property type) would be great additions! Thanks for the conversation

[layomia]

We can also, or alternatively, consider a property on JsonConstructorAttribute, that indicates no restriction between the ctor parameter type and the immutable property type. This would allow them to be two arbitrarily different types, basically an "I know what I'm doing mode". It would still be required for their CLR names to match.

We would likely still need to support the "mapping nullable value-type ctor args to immutable properties of a non-nullable version of the type" case by default - #44428 (comment).

[layomia]

Co-assigning @GabeDeBacker to provide the implementation for this feature, as discussed offline.

[GabeDeBacker]

I did find an example of where a property type is not assignable from the constructor argument type.
You cannot construct a SecureString from a string.

public class ClassThatStoresSecureStrings
{
    using System;
    using System.Security;

   public ClassThatStoresSecureStrings(string userId, string password)
   {
            // This code repeats for password.
            this.UserId = new SecureString();
            foreach (var ch in userId)
            {
                this.UserId.AppendChar(ch);
            }

            this.UserId.MakeReadOnly();
   }

   public SecureString UserId { get; }
   public SecureString Password {get; }
}

[eiriktsarpalis]

AFAIK you're the first to bring up that concern. I think it's reasonable to factor into a separate issue since it's of much smaller scope.

[mathieubergouniouxcab]

AFAIK you're the first to bring up that concern. I think it's reasonable to factor into a separate issue since it's of much smaller scope.

I found 8+ entries in your bug tracker related to the error "Each parameter in constructor (...) must bind to a field on deserialization". Devs so obsessed with grinding the issues one by one that they forgot it's 1,000 different flavours of the same issue. https://i.imgur.com/B0St5fv.png

[eiriktsarpalis]

Care to share the issues that specifically requests this concern?

Something that no one mentioned : If the error message could at least indicate what parameter caused the issue then that would be the bare minimum.

[mathieubergouniouxcab]

Care to share the issues that specifically requests this concern?

The google query I used :

site:https://github.com/dotnet/runtime/issues/ Each parameter in constructor must bind to an object property or field on deserialization

I only clicked on the 8 first ones but there's 20+.
Here are some metrics to evaluate the impact :

• Most of them complain that the typing is too strict (e.g. fails to recognize that a List can be deserialized as a IEnumerable )
• very few mention issues witht he parameter's name. But imho it's a classic case of survivor bias, as in : we see only the "valid" issues that made it to the tracker. We don't see the stupid ones, e.g. typos in parameters names, even though they took just as long to spot and fix.

[eiriktsarpalis]

What is the point you are trying to make? Most of the issues in that query are closed which means that either the error string appears tangentially or the issue has been closed as duplicate of the very issue we are discussing in right now.

[mathieubergouniouxcab]

Precisely. They are duplicates. Which means the same issue keeps coming back like a boomerang because users are not sure if it's by design and report it as a bug, when a clearer message would help.

So anyways, here's the issue created :
#88048

[eiriktsarpalis]

Precisely. They are duplicates. Which means the same issue keeps coming back like a boomerang because devs are not sure if it's by design, when a clearer message would help.

What exactly are you proposing? That we stop closing duplicates so that it somehow makes the issue more pronounced or discoverable? That never works in practice. When we close a duplicate we link to the original issue and encourage folks to upvote and contribute there.

[mathieubergouniouxcab]

What exactly are you proposing? That we stop closing duplicates ?

No, I'm not suggesting anything. I think a misunderstanding happened at some point in the recent messages. Let's move on: I'm done with this topic, I'm interested only in the error message, as you saw in #88048 . I'll stop polluting this thread now. Thanks for the time and efforts! <3

[Greenscreener]

Hi!

Since this has been silent for over a year, I thought I might chip in with my current use case.

I'm currently working with an API, that looks like this:

{
  "content": { ... },
  "type": "text"
}

Where the type property determines the type of the object stored in the content property. This is somewhat similar to what is possible with polymorphic serialization type discriminators, just that the type is outside the object instead of inside it.

I'm trying to implement serialization of this type by adding a JsonConstructor, which receives the content property as a JsonObject. A suitable type is then picked using the type property and a JsonSerializer.Serialize<Type>(obj) is called to serialize the type. Here is the relevant snippet:

	public record Event(
		EventContent content,
		string type
	) {
		[JsonConstructor]
		public Event(JsonObject content, string type) : this(EventContent.FromJSON(type, content), type) {}
	};

This is exactly the case where I'd need a "I know what I'm doing" mode, since the parameters will never, by design, match.

I find somewhat fascinating that adding a toggle like is so much work it couldn't be done in two versions, but I digress.

PS: If anyone has any idea for a workaround, it would help me a lot. If you're interested, here's the whole file where the issue lies: https://gitlab.com/Greenscreener/matrix-dotnet/-/blob/master/MatrixApi.cs?ref_type=heads

[Greenscreener]

FYI: I've found a workaround, but I find it incredibly ugly:

	public record Event {
		[JsonIgnore]
		public EventContent content {get;}
		[JsonPropertyName("content")]
		public JsonObject _content {init; private get;}
		public string type {get;}
		
		public Event(JsonObject _content, string type) {
			this.content = EventContent.FromJSON(type, _content);
			this._content = _content;
			this.type = type;
		}
	}

EDIT: This does not work for serialization, oh well, back to the drawing board.

[Timovzl]

@Greenscreener To complete the workaround to include serialization, could you perhaps use a calculated _content getter that serializes content based on type? Very annoyingly, you'd lose the outer serialization options, but it could be better than nothing.

[Greenscreener]

@Timovzl I ended up writing a huge custom converter that handles my entire use case, FWIW, here's the source: https://gitlab.com/Greenscreener/matrix-dotnet/-/blob/master/PolymorphicJson.cs?ref_type=heads

[terrajobst]

@Greenscreener

EDIT: This does not work for serialization, oh well, back to the drawing board.

A custom converter (like what you have now) is probably the cleanest solution. But for the record, you could achieve your scenario also like this:

{
  "content": { /* ... */ },
  "type": "text"
}

If the type is read-only/immutable the following would work:

class DataHolder
{
    public DataHolder(string type, JsonObject content)
    {
        Type = type;
        ContentJson = content;
        Content = GetObjectForJson(type, content);
    }

    public string Type { get; }

    [JsonProperty("content")]
    [EditorBrowsable(EditorBrowsableState.Never)]
    public JsonObject ContentJson { get; }

    [JsonIgnore]
    public object Content { get; }

    private object GetObjectForJson(string type, JsonObject content)
    {
        // custom logic
    }
}

If the type is meant to be mutable, I'd go with something like this:

class DataHolder
{
    private string _type;
    private JsonObject _contentJson;
    private object _content;

    [JsonConstructor]
    public DataHolder(string type, JsonObject content)
    {
        _type = type;
        _contentJson = content;
        _content = GetObjectForJson(type, content);
    }

    public DataHolder(object content)
    {
        _type = GetTypeForObject(content);
        _contentJson = GetJsonForObject(content);
        _content = content;
    }

    public string Type => _type;

    [JsonProperty("content")]
    [EditorBrowsable(EditorBrowsableState.Never)]
    public JsonObject ContentJson => _contentJson;

    [JsonIgnore]
    public object Content
    {
        get;
        set 
        {
            _type = GetTypeForObject(value);
            _contentJson = GetObjectForJson(value);
            _content = value;
        }
    }

    private static object GetObjectForJson(string type, JsonObject content)
    {
        // custom logic
    }

    private static string GetTypeForObject(object content)
    {
        // custom logic 
    }

    private static JsonObject GetJsonForObject(object content)
    {
        // custom logic 
    }
}

[Timovzl]

Without a custom converter, the issue remains that the (de)serialization logic will not receive the JsonSerializerOptions being used by the outer serializer. When serializing, the output format of the inner JSON may then mismatch in casing, enum representation, custom conversions, etc. When deserializing, the result may be incorrect or incomplete.

[terrajobst]

Without a custom converter, the issue remains that the (de)serialization logic will not receive the JsonSerializerOptions being used by the outer serializer. When serializing, the output format of the inner JSON may then mismatch in casing, enum representation, custom conversions, etc. When deserializing, the result may be incorrect or incomplete.

That's right. Generally speaking, here is how to think about it:

1. If you want quick way to tweak the result types by wrapping or restructuring the data, the property approach will work. However, this generally assumes that you either need custom serialization logic that is independent of the outer serialization (e.g. a custom format) or you just don't like the managed presentation (e.g. instead of an array of objects you want to return some custom type that the serializer can't easily construct).

2. If you want to recurse into the serializer, then you almost always want a custom converter. It's the only way to access the serialization options and/or ask nested data to go through the same serializer.

[fabio-s-franco]

Is there still a point to this thread? It seems a bit of a waste of time. People continue to pitch in, spending their time on something that gets no serious attention. It's better to just mark it as not planned rather than pushing for 4 years. I have done what any sane person that touched this would do: Look for a better alternative.
I have abandoned the usage of this API for over a year now, it is not worth the frustration and the amount of friction it creates within teams.
There were a bunch of enhancements to json serialization and deserialization, but what is the point if usability is not addressed. If people are pushed away from using it, then enhancements are made for a niche that stayed with it. I don't know how adoption is nowadays, and keep watching this thread, but I my feeling is that it hasn't gotten more popular since.

[eiriktsarpalis]

While the existing restriction is well-intentioned in that it ensures that users don't write models that are not round-tripable, it's pretty painful in the cases where it doesn't work. We could try to make the matching algorithm smarter so that similarly shaped types are considered equivalent, however I doubt this would be addressing the problem completely. There are many cases where a mismatch could be intentional (e.g. because the type is only used for deserialization).

I think the best course of action is to expose a switch that disables constructor parameter matching for a given type, or globally.

API Proposal

namespace System.Text.Json.Serialization;

public partial class JsonConstructorAttribute
{
+   public bool DisableParameterMatching { get; set; }
}

And for the global variant:

namespace System.Text.Json;

public partial class JsonSerializerOptions
{
+   public bool DisableConstuctorParameterMatching { get; set; }
}

namespace System.Text.Json.Serialization;

public partial class JsonSourceGenerationOptionsAttribute
{
+   public bool DisableConstuctorParameterMatching { get; set; }
}

API Usage

JsonSerializer.Deserialize<MyPoco>("""{"x" : 1, "y" : 2 }"""); // Succeeds

public class MyPoco
{
     [JsonConstructor(DisableParameterMatching = true)]
     public MyPoco(int x, int y) { }
}

[raffaeler]

@eiriktsarpalis This was indeed the initial proposal:
Please ensure to add a global option in the JsonSerializerOptions (the proposed property was ConstructorUseRelaxedPropertyMatching ) so that we can use it without attribute decoration.
Please also consider this comment.

[terrajobst]

@eiriktsarpalis

While the existing restriction is well-intentioned in that it ensures that users don't write models that are not round-tripable

Could you explain what this means? Naively, I'd say that as long as there is an implicit conversion from the property type to the parameter type round-tripping is guaranteed by the constructor. For example, if the parameter is nullable and the property isn't or if the parameter is IEnumerable<T> but the property is T[] or ImmutableArray<T>. There is, of course, cost to check for that, but the check would only execute where it currently fails the type comparison checks, so it won't regress the most common cases.

Personally, I'd hate to see an opt-in for this feature. I guess an opt-out would be reasonable, but it seems weird given that the alternative is throwing an error.

[eiriktsarpalis]

While the existing restriction is well-intentioned in that it ensures that users don't write models that are not round-tripable

Could you explain what this means? Naively, I'd say that as long as there is an implicit conversion from the property type to the parameter type round-tripping is guaranteed by the constructor.

I think it's reasonable to relax the restriction on the basis of a type relationship (allow constructor parameters that are assignable/convertible from the corresponding property parameter, on the basis of the principle that valid deserialization inputs should be a superset of the valid serialization outputs). My point though is that the problem extends beyond the narrow scope of subtyping, for example one might reasonably expect that this is a valid model:

public class MyPoco(IEnumerable<IEnumerable<string>> tokens)
{
    public ImmutableArray<ImmutableArray<string>> Tokens { get; } = tokens.Select(x => x.ToImmutableArray()).ToImmutableArray();
}

In which case there is no type relationship, even though structurally speaking the parameter and property types are equivalent.

[terrajobst]

for example one might reasonably expect that this is a valid model:

That's fair, but I think this would violate our stated goal because it can't be roundtripped (and as far as I can tell nobody is asking for that yet).

I suspect our solution probably needs to support more than Type.IsAssignableFrom, specifically handle conversion operators.

[eiriktsarpalis]

That's fair, but I think this would violate our stated goal because it can't be roundtripped

Why not? The shapes of the two types are equivalent.

(and as far as I can tell nobody is asking for that yet).

We've had a few duplicates historically asking for variations of the same thing. I'm not saying we should actually build this, the point is we should have an "I know what I'm doing" mode that disables matching altogether.

suspect our solution probably needs to support more than Type.IsAssignableFrom, specifically handle conversion operators.

I think that's fair, although it will be tricky to implement full parity with language semantics using reflection.

[terrajobst]

I'm not saying we should actually build this, the point is we should have an "I know what I'm doing" mode that disables matching altogether.

I think I'm missing how this would work? It seems if the values the constructor accepts can't be derived in an automatic fashion from the properties, then round tripping will fail. Having an expert mode wouldn't change that, right?

[eiriktsarpalis]

That's correct, however oftentimes this might be desirable (e.g. because the type is binding model so we only care about deserialization). At the same time, parameter matching itself doesn't offer any strong guarantees wrt round tripping properties, there are many reasons why it might still fail for a particular model (abstract types, non-public constructors).

[LeszekKalibrate]

4 years later and still simple deserialization, assign value for nullable param in ctor when there is non nullable field/property is not working.

[NullQubit]

4 years later and still simple deserialization, assign value for nullable param in ctor when there is non nullable field/property is not working.

Not only that, this is also not working, which is extremely surprising to me

public class Book
{
    public IReadOnlyList<string> Pages { get; }

    public Book(IEnumerable<string> pages)
    {
        Pages = pages.ToList().AsReadOnly();
    }
}