System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

frankbuckley · 2020-09-10T13:08:47Z

While evaluating .NET 5.0 (Preview 8), I have encountered a problem with the JsonSerializer deserializing a particular payload, which works fine with .NET Core 3.1.

Description

Specifically, the payload is the response from a call to the Shopify Customer Search API. If we call this with a limit of 200 results, it returns a ~330KB response body and an exception is thrown. If we call this same API with a limit of 100 results, deserialization of the response completes successfully.

I have managed to reproduce the error, reading the response data from a file. It appears that the error only occurs if JsonSerializerOptions.IgnoreNullValues = true and deserializing(async) from a stream. If the Json is read into memory and JsonSerializer.Deserialize is called, it works fine.

I have tried reproducing the error with another large payload without success, so it seems to be something specific to this Json data (and this configuration). Unfortunately, it includes personal data so I cannot share it publicly, but I would be happy to find a way to help debug the problem.

Configuration

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-preview.8.20417.9
 Commit:    fc62663a35

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-preview.8.20417.9\

Host (useful for support):
  Version: 5.0.0-preview.8.20407.11
  Commit:  bf456654f9

.NET SDKs installed:
  3.1.402 [C:\Program Files\dotnet\sdk]
  5.0.100-preview.8.20417.9 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0-preview.8.20414.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0-preview.8.20407.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0-preview.8.20411.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Regression?

Yes - this works with .NET Core 3.1.

Other information

The exception thrown is:

System.Text.Json.JsonException
  HResult=0x80131500
  Message=The JSON value could not be converted to System.String. Path: $.customers[135].default_address.province | LineNumber: 0 | BytePositionInLine: 229465.
  Source=System.Text.Json
  StackTrace:
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.<ReadAsync>d__19`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at JsonDotnet5Repro.Program.<Main>d__4.MoveNext() in C:\dev\frankbuckley\JsonDotnet5Repro\JsonDotnet5Repro\Program.cs:line 43

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
InvalidOperationException: Cannot get the value of a token type 'StartObject' as a string.

InnerException stack trace:

   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.Serialization.Converters.StringConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

Repro (excluding data):

using System;
using System.Collections.Generic;
using System.IO;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;

namespace JsonDotnet5Repro
{
    class Program
    {
        private static readonly JsonSerializerOptions s_serializerOptions = new JsonSerializerOptions
        {
            IgnoreNullValues = true // works if set to false
        };

        static async Task Main(string[] args)
        {
            // Read into memory (always works)

            var json = File.ReadAllText("response.json");
            var result1 = JsonSerializer.Deserialize<CustomerCollectionResponse>(json, s_serializerOptions);

            Console.WriteLine($"Reading Json in memory - deserialized {result1.Customers.Count} customer records.");

            // Read from stream (fails if IgnoreNullValues = true)

            var stream = File.OpenRead("response.json");
            var result2 = await JsonSerializer.DeserializeAsync<CustomerCollectionResponse>(stream, s_serializerOptions);

            Console.WriteLine($"Reading Json from stream - deserialized {result2.Customers.Count} customer records.");
        }
    }

    public class CustomerCollectionResponse
    {
        [JsonPropertyName("customers")]
        public List<Customer>? Customers { get; set; }
    }

    public class Address
    {
        [JsonPropertyName("first_name")]
        public string? FirstName { get; set; }

        [JsonPropertyName("address1")]
        public string? Address1 { get; set; }

        [JsonPropertyName("phone")]
        public string? Phone { get; set; }

        [JsonPropertyName("city")]
        public string? City { get; set; }

        [JsonPropertyName("zip")]
        public string? Zip { get; set; }

        [JsonPropertyName("province")]
        public string? Province { get; set; }

        [JsonPropertyName("country")]
        public string? Country { get; set; }

        [JsonPropertyName("last_name")]
        public string? LastName { get; set; }

        [JsonPropertyName("address2")]
        public string? Address2 { get; set; }

        [JsonPropertyName("company")]
        public string? Company { get; set; }

        [JsonPropertyName("latitude")]
        public float? Latitude { get; set; }

        [JsonPropertyName("longitude")]
        public float? Longitude { get; set; }

        [JsonPropertyName("name")]
        public string? Name { get; set; }

        [JsonPropertyName("country_code")]
        public string? CountryCode { get; set; }

        [JsonPropertyName("province_code")]
        public string? ProvinceCode { get; set; }
    }

    public class Customer
    {
        [JsonPropertyName("id")]
        public long Id { get; set; }

        [JsonPropertyName("email")]
        public string? Email { get; set; }

        [JsonPropertyName("accepts_marketing")]
        public bool AcceptsMarketing { get; set; }

        [JsonPropertyName("created_at")]
        public DateTimeOffset? CreatedAt { get; set; }

        [JsonPropertyName("updated_at")]
        public DateTimeOffset? UpdatedAt { get; set; }

        [JsonPropertyName("first_name")]
        public string? FirstName { get; set; }

        [JsonPropertyName("last_name")]
        public string? LastName { get; set; }

        [JsonPropertyName("orders_count")]
        public int OrdersCount { get; set; }

        [JsonPropertyName("state")]
        public string? State { get; set; }

        [JsonPropertyName("total_spent")]
        public string? TotalSpent { get; set; }

        [JsonPropertyName("last_order_id")]
        public long? LastOrderId { get; set; }

        [JsonPropertyName("note")]
        public string? Note { get; set; }

        [JsonPropertyName("verified_email")]
        public bool VerifiedEmail { get; set; }

        [JsonPropertyName("multipass_identifier")]
        public string? MultipassIdentifier { get; set; }

        [JsonPropertyName("tax_exempt")]
        public bool TaxExempt { get; set; }

        [JsonPropertyName("tags")]
        public string? Tags { get; set; }

        [JsonPropertyName("last_order_name")]
        public string? LastOrderName { get; set; }

        [JsonPropertyName("default_address")]
        public Address DefaultAddress { get; set; }

        [JsonPropertyName("addresses")]
        public IList<Address> Addresses { get; set; }
    }
}

I found a similar issue (#41604) but this error differs in that it is occurring in an ordinary Console app on Windows and also running in Linux (Ubuntu Fedora) containers.

The text was updated successfully, but these errors were encountered:

am11 · 2020-09-10T14:34:21Z

On macOS, I generated a 13M of /tmp/response.json with:

$ node -e "fs.writeFileSync('/tmp/response.json', JSON.stringify({customers: [...Array(1000000).keys()].map(x=>({id: x}))}))"

and ran your code sample (adjusting response.json as /tmp/response.json) in a console app generated with SDK 5.0.0-rc.2.20458.14 (and host 5.0.100-rc.2.20459.18), it seems to to be working fine:

$ ~/.dotnet5/dotnet --version
5.0.100-rc.2.20459.18

$ ~/.dotnet5/dotnet run
Reading Json in memory - deserialized 1000000 customer records.
Reading Json from stream - deserialized 1000000 customer records.

frankbuckley · 2020-09-10T14:56:44Z

@am11 - thanks - I have checked and I can reproduce on MacOS with RC2. The error is consistently replicable, but is sensitive to the data. I spent some time trying to find an example of data that does not include private information and reproduces the error, but could not find anything else that does it.

PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro> dotnet --info
.NET SDK (reflecting any global.json):
Version:   5.0.100-rc.2.20460.9
Commit:    c8ff60c134

Runtime Environment:
OS Name:     Mac OS X
OS Version:  10.15
OS Platform: Darwin
RID:         osx.10.15-x64
Base Path:   /usr/local/share/dotnet/sdk/5.0.100-rc.2.20460.9/

Host (useful for support):
Version: 5.0.0-rc.2.20458.14
Commit:  482494f9ec

.NET SDKs installed:
3.1.401 [/usr/local/share/dotnet/sdk]
3.1.402 [/usr/local/share/dotnet/sdk]
5.0.100-preview.8.20417.9 [/usr/local/share/dotnet/sdk]
5.0.100-rc.2.20460.9 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-preview.8.20414.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-rc.2.20459.20 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-preview.8.20407.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-rc.2.20458.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download
PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro> dotnet run
Reading Json in memory - deserialized 200 customer records.
Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.customers[135].default_address.province | LineNumber: 0 | BytePositionInLine: 229465.
---> System.InvalidOperationException: Cannot get the value of a token type 'StartObject' as a string.
at System.Text.Json.Utf8JsonReader.GetString()
at System.Text.Json.Serialization.Converters.StringConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
--- End of inner exception stack trace ---
at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
at JsonDotnet5Repro.Program.Main(String[] args) in /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro/Program.cs:line 29
at JsonDotnet5Repro.Program.<Main>(String[] args)
PS /Users/frankbuckley/dev/frankbuckley/JsonDotnet5Repro>

tmds · 2020-09-11T11:57:39Z

It's possible this is fixed by #42089.

am11 · 2020-09-11T13:14:36Z

With 125M of response.json with 1M elements in customers list, the memory API only reads 100K (result1.Customers.Count), while stream API returns null (no exception, we get nullref from the next line result2.Customers.Count). I am not sure if this is expected behavior from either of the APIs. 🤔

frankbuckley · 2020-09-11T15:00:35Z

Here is a repro with fake data: https://github.com/frankbuckley/net5-json-repro

Running gives results similar to:

PS C:\dev\frankbuckley\net5-json-repro> dotnet run
0000: Reading Json from stream - deserialized 500 customer records.
0001: Reading Json from stream - deserialized 500 customer records.
0002: Reading Json from stream - deserialized 500 customer records.
0003: Reading Json from stream - deserialized 500 customer records.
0004: Reading Json from stream - deserialized 500 customer records.
0005: Reading Json from stream - deserialized 500 customer records.
0006: Reading Json from stream - deserialized 500 customer records.
0007: Reading Json from stream - deserialized 500 customer records.
0008: Reading Json from stream - deserialized 500 customer records.
0009: Reading Json from stream - deserialized 500 customer records.
0010: Reading Json from stream - deserialized 500 customer records.
Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.String. Path: $.customers[66].addresses[3].province | LineNumber: 0 | BytePositionInLine: 147609.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'EndObject' as a string.
   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Program.Main(String[] args) in C:\dev\frankbuckley\net5-json-repro\Program.cs:line 47
   at Program.<Main>(String[] args)

To do this reliably, the .RuleFor(a => a.Province, f => f.Address.State().OrNull(f, 0.7f)) seems to be required (at least, I have not yet seen the error without it).

devsko · 2020-09-12T13:13:49Z

This happens, when the buffered data ends exactly while reading a null token.
When the next block of data is available, the state continuation mechanism stops (after entering the first frame) at this condition in JsonPropertyInfo<T>.ReadJsonAndSetMember() which I think should also include state.IsContinuation

237                if (!isNullToken || !IgnoreDefaultValuesOnRead || !Converter.CanBeNull)

Will try to fix it without regression or at least add a simple repro.

* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx

layomia · 2020-09-16T23:45:41Z

Re-opening for consideration in .NET 5.

* Repro dotnet#42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from dotnet#42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx

* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx Co-authored-by: devsko <devsko@users.noreply.github.com>

layomia · 2020-09-18T05:02:39Z

Fixed for .NET 5 in #42359.

* Repro #42070 * formatting * namespace * Fix * never forget the header * More tests - Test continuation at every position inside the tested object - Many member with primitive and nullable types - One more level of nested object - All combinations of class/struct for tested and nested object - tested and nested object with parametrized ctor for some properties * Addressed feedback * refactoring * Test with original repro data from #42070 * custom converter to ensure the padding is written in front of the tested object * rename * test data moved to Strings.resx * Using test data from SR * Generalize continuation tests for payloads of any length Tweak the payload and expect `JsonException` * merge * Test deserialize with Utf8JsonReader and ReadOnlySequence * Again with value typed nested object * Add tests for splitted whitespaces * Addressed feedback Added dictionary test * Validate line and position of failure in tweaked payloads more tweaks * Fixed comment Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com>

Dotnet-GitSync-Bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Sep 10, 2020

layomia removed the untriaged New issue has not been triaged by the area owner label Sep 10, 2020

layomia added this to the 5.0.0 milestone Sep 10, 2020

ericstj added regression-from-last-release blocking-release bug labels Sep 10, 2020

devsko added a commit to devsko/runtime that referenced this issue Sep 12, 2020

Repro dotnet#42070

d004c08

devsko mentioned this issue Sep 12, 2020

Fix continuation of JSON deserialization #42158

Merged

devsko added a commit to devsko/runtime that referenced this issue Sep 16, 2020

Test with original repro data from dotnet#42070

329bc83

layomia closed this as completed in #42158 Sep 16, 2020

layomia reopened this Sep 16, 2020

layomia mentioned this issue Sep 17, 2020

[release/5.0-rc2] Prevent JSON async deserialization errors when partial buffer ends in the middle of null token (#42158) #42359

Merged

layomia closed this as completed Sep 18, 2020

ghost locked as resolved and limited conversation to collaborators Dec 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

frankbuckley commented Sep 10, 2020

am11 commented Sep 10, 2020

frankbuckley commented Sep 10, 2020

tmds commented Sep 11, 2020

am11 commented Sep 11, 2020

frankbuckley commented Sep 11, 2020

devsko commented Sep 12, 2020 •

edited

Loading

layomia commented Sep 16, 2020

layomia commented Sep 18, 2020

System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

System.Text.Json.JsonException deserializing large Json stream with .NET 5.0 Preview 8 (but not 3.1) #42070

Comments

frankbuckley commented Sep 10, 2020

Description

Configuration

Regression?

Other information

am11 commented Sep 10, 2020

frankbuckley commented Sep 10, 2020

tmds commented Sep 11, 2020

am11 commented Sep 11, 2020

frankbuckley commented Sep 11, 2020

devsko commented Sep 12, 2020 • edited Loading

layomia commented Sep 16, 2020

layomia commented Sep 18, 2020

devsko commented Sep 12, 2020 •

edited

Loading