Cannot deserialize an immutable collection

[asik]

Description

I'm trying to implement a type that is very similar to System.Collections.Immutable.ImmutableArray; actually it is based entirely on that type, it only overrides equality. So far it looks like this:

    public readonly struct Block<T> : 
        IReadOnlyList<T>, 
        IEquatable<Block<T>>
    {
        readonly ImmutableArray<T> _arr;
        public Block(IEnumerable<T> elems) => _arr = elems.ToImmutableArray();
        // etc. 
    }

I'm running into an issue trying to deserialize an instance of that type:

    [Fact]
    void Test()
    {
        var original = Block.Create(1, 2, 3);
        var serialized = JsonSerializer.Serialize(original);
        var deserialized = JsonSerializer.Deserialize<Block<int>>(serialized); // <-- exception here

        Assert.Equal(original, deserialized);
    }
/*
  Message: 
System.NotSupportedException : The collection type 'ValueCollections.Block`1[System.Int32]' is abstract, an interface, or is read only, and could not be instantiated and populated. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
---- System.NotSupportedException : The collection type 'ValueCollections.Block`1[System.Int32]' is abstract, an interface, or is read only, and could not be instantiated and populated.

  Stack Trace: 
ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex)
ThrowHelper.ThrowNotSupportedException_CannotPopulateCollection(Type type, Utf8JsonReader& reader, ReadStack& state)
ICollectionOfTConverter`2.CreateCollection(Utf8JsonReader& reader, ReadStack& state, JsonSerializerOptions options)
JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
JsonSerializationTests.Try() line 19
*/

The exact same test works perfectly fine with ImmutableArray. I read that I should implement ICollection<T> (or IList<T>), so I did, but with IsReadOnly returning true (my type is immutable). JsonSerializer doesn't like this. But ImmutableArray also implements it by returning true. So what's going on there?

I found that when deserializing ImmutableArray, an ImmutableEnumerableOfTConverterWithReflection is created. This appears to govern when this type is created: unfortunately, does so by checking that the type is in "System.Collections.Immutable" and is one of a list of hardcoded types from that assembly. So I'm out of luck.

I finally tried adding a [JsonConstructor] attribute over the constructor taking an IEnumerable<T>, but no dice. I also tried exposing the underlying ImmutableArray via a [JsonInclude] property, but that doesn't help, JsonSerializer serializes this as an array, not a json object.

There is support for deserializing immutable types, but this just does not seem to encompass collections.

Am I missing anything or the support for this is simply not there?

Reproduction Steps

You can use this nuget https://www.nuget.org/packages/ValueCollections.Block/0.0.1-alpha and just try to serialize and deserialize a value of the type, eg:

        var original = Block.Create(1, 2, 3);
        var serialized = JsonSerializer.Serialize(original);
        var deserialized = JsonSerializer.Deserialize<Block<int>>(serialized);

Expected behavior

It should be possible to deserialize an immutable collection type.

Actual behavior

It's not possible to deserialize an immutable collection type unless it is one of the hardcoded ones in System.Collections.Immutable.

Regression?

No response

Known Workarounds

No response

Configuration

The type itself is on .NET Standard 2.0.
I was testing serialization from a .NET 6 program on Windows 64-bit, Any CPU.

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm trying to implement a type that is very similar to System.Collections.Immutable.ImmutableArray; actually it is based entirely on that type, it only overrides equality. So far it looks like this:

    public readonly struct Block<T> : 
        IReadOnlyList<T>, 
        IEquatable<Block<T>>
    {
        readonly ImmutableArray<T> _arr;
        public Block(IEnumerable<T> elems) => _arr = elems.ToImmutableArray();
        // etc. 
    }

I'm running an into trying to deserialize an instance of that type:

    [Fact]
    void Test()
    {
        var original = Block.Create(1, 2, 3);
        var serialized = JsonSerializer.Serialize(original);
        var deserialized = JsonSerializer.Deserialize<Block<int>>(serialized); // <-- exception here

        Assert.Equal(original, deserialized);
    }
/*
  Message: 
System.NotSupportedException : The collection type 'ValueCollections.Block`1[System.Int32]' is abstract, an interface, or is read only, and could not be instantiated and populated. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
---- System.NotSupportedException : The collection type 'ValueCollections.Block`1[System.Int32]' is abstract, an interface, or is read only, and could not be instantiated and populated.

  Stack Trace: 
ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex)
ThrowHelper.ThrowNotSupportedException_CannotPopulateCollection(Type type, Utf8JsonReader& reader, ReadStack& state)
ICollectionOfTConverter`2.CreateCollection(Utf8JsonReader& reader, ReadStack& state, JsonSerializerOptions options)
JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
JsonSerializationTests.Try() line 19
*/

The exact same test works perfectly fine with ImmutableArray. I read that I should implement ICollection<T> (or IList<T>), so I did, but with IsReadOnly returning true (my type is immutable). JsonSerializer doesn't like this. But ImmutableArray also implements it by returning true. So what's going on there?

I found that when deserializing ImmutableArray, an ImmutableEnumerableOfTConverterWithReflection is created. This appears to govern when this type is created: unfortunately, does so by checking that the type is in "System.Collections.Immutable" and is one of a list of hardcoded types from that assembly. So I'm out of luck.

I finally tried adding a [JsonConstructor] attribute over the constructor taking an IEnumerable<T>, but no dice. I also tried exposing the underlying ImmutableArray via a [JsonInclude] property, but that doesn't help, JsonSerializer serializes this as an array, not a json object.

There is support for deserializing immutable types, but this just does not seem to encompass collections.

Am I missing anything or the support for this is simply not there?

Reproduction Steps

You can use this nuget https://www.nuget.org/packages/ValueCollections.Block/0.0.1-alpha and just try to serialize and deserialize a value of the type, eg:

        var original = Block.Create(1, 2, 3);
        var serialized = JsonSerializer.Serialize(original);
        var deserialized = JsonSerializer.Deserialize<Block<int>>(serialized);

Expected behavior

It should be possible to deserialize an immutable collection type.

Actual behavior

It's not possible to deserialize an immutable collection type unless it is one of the hardcoded ones in System.Collections.Immutable.

Regression?

No response

Known Workarounds

No response

Configuration

The type itself is on .NET Standard 2.0.
I was testing serialization from a .NET 6 program on Windows 64-bit, Any CPU.

Other information

No response

Author:	asik
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[gregsdennis]

@asik you've explicitly marked your type as readonly. Is that necessary for your application? Could you make it semantically read-only?

The error message says that the serializer can't deserialize read-only types.

[asik]

Hi @gregsdennis , I wouldn't mind if removing readonly before struct was a workaround, but I still get the same error. On a related note, if I implement ICollection to return IsReadOnly = false, this is going to try to call ICollection<T>.Add which I can't support (actually, I'd rather not implement ICollection at all since it would violate LSP for my type).

[eiriktsarpalis]

Duplicate of #38514. Deserialization for arbitrary collection types is not supported since the serializer cannot know how the type is meant to be created or populated (as the error message suggests). We would consider addressing this via #63791.

A quick workaround is to author your own custom converter for the collection type.