JsonSerializer.DeserializeAsyncEnumerable fails on a valid JSON stream that DeserializeAsync handles correctly

[rampaa]

Description

When deserializing a JSON stream, JsonSerializer.DeserializeAsyncEnumerable<T> throws a System.Text.Json.JsonException on data that appears to be perfectly valid. However, using the JsonSerializer.DeserializeAsync<T> method on the exact same stream succeeds without error.

The failure does not appear to be inherent to the JSON content itself; a minified version of the same data, or a smaller subset of the records, parses correctly with DeserializeAsyncEnumerable.

Reproduction Steps

1. Download Test.json.

2. Try deserializing it using the following code:

   FileStream fileStream = File.OpenRead(jsonFilePath);
   await using (fileStream.ConfigureAwait(false))
   {
       IAsyncEnumerable<ReadOnlyMemory<JsonElement>> jsonElementsEnumerable =
           JsonSerializer.DeserializeAsyncEnumerable<ReadOnlyMemory<JsonElement>>(fileStream);
   }

   You'll see the following exception:

   System.Text.Json.JsonException: 'The JSON value could not be converted to System.ReadOnlyMemory`1[System.Text.Json.JsonElement]. Path: $[1] | LineNumber: 2950 | BytePositionInLine: 10.'
   

3. Now try deserializing it using the following code:

   FileStream fileStream = File.OpenRead(jsonFilePath);
   await using (fileStream.ConfigureAwait(false))
   {
       ReadOnlyMemory<ReadOnlyMemory<JsonElement>> jsonElementsRom =
           await JsonSerializer
               .DeserializeAsync<ReadOnlyMemory<ReadOnlyMemory<JsonElement>>>(fileStream)
               .ConfigureAwait(false);
   }

   This time, it works without any errors.

4. Finally, try deserializing the minified version of the same JSON file (TestMinified.json) using the same code as in step 2. Notice that it also works without any errors.

Expected behavior

DeserializeAsyncEnumerable should successfully parse the JSON without throwing any exceptions. Whether the JSON file is minified or not should not affect the behavior of DeserializeAsyncEnumerable.

Actual behavior

DeserializeAsyncEnumerable fails to parse the JSON file with the following exception, and its behavior is affected by whether the JSON file is minified:

```
System.Text.Json.JsonException: 'The JSON value could not be converted to System.ReadOnlyMemory`1[System.Text.Json.JsonElement]. Path: $[1] | LineNumber: 2950 | BytePositionInLine: 10.'
```

Regression?

No response

Known Workarounds

Using DeserializeAsync instead of DeserializeAsyncEnumerable

Configuration

Which version of .NET is the code running on?

.NET 9

What OS and version, and what distro if applicable?

Windows 10 22H2

What is the architecture (x64, x86, ARM, ARM64)?

x64

Other information

No response

[eiriktsarpalis]

I can't reproduce using .NET 9 and your sample data. Note that your repro isn't actually consuming the IAE so I updated it to this:

string jsonFilePath = "Test.json";
FileStream fileStream = File.OpenRead(jsonFilePath);
await using (fileStream.ConfigureAwait(false))
{
    IAsyncEnumerable<ReadOnlyMemory<JsonElement>> jsonElementsEnumerable =
        JsonSerializer.DeserializeAsyncEnumerable<ReadOnlyMemory<JsonElement>>(fileStream);

    int i = 0;
    await foreach (ReadOnlyMemory<JsonElement> jsonElement in jsonElementsEnumerable.ConfigureAwait(false))
    {
        i++;
    }

    Console.WriteLine($"Total JSON elements processed: {i}");
}

Yielding

Total JSON elements processed: 51

[rampaa]

@eiriktsarpalis sorry for the incomplete code example. Here's a repro project that should hopefully allow you to repro it:
IAsyncEnumerableBug.zip

Here's the dump file in case it's helpful:
https://limewire.com/d/iBmD7#sIjtQFBpFX

[eiriktsarpalis]

I can reproduce. It seems to impact all async deserialization methods and not just IAsyncEnumerable. Minimal repro:

using System.Text;
using System.Text.Json;

string json = File.ReadAllText("Test.json");
using MemoryStream fileStream = new(Encoding.UTF8.GetBytes(json));
var result = await JsonSerializer.DeserializeAsync<List<Memory<JsonElement>>>(fileStream);

int i = 0;
foreach (var element in result!)
{
    i++;
}

Console.WriteLine($"Total JSON elements processed: {i}");

What's weird is that I can only reproduce when the inner collection is either of type Memory<JsonElement> or ReadOnlyMemory<JsonElement>. It doesn't seem to repro with other collection types. I also wasn't able to create a smaller JSON payload reproducing the issue, nor was I able to induce the error using synthetic data.

Test.json

[rampaa]

Unfortunately, I have several JSON files that trigger this error. Test.json was my best attempt at isolating and minimizing the issue. I'm happy to share the other files as well if that would help in resolving the problem.

[eiriktsarpalis]

If you could somehow share anything that is smaller, that would be very helpful.

[rampaa]

Unfortunately, I wasn’t able to create a smaller JSON file that reproduces the issue.

You’ve probably already noticed this, but interestingly, even just changing the line endings in Test.json from LF to CRLF prevents the issue from occurring. I'm not sure if that helps narrow down the root cause, but I thought it was worth mentioning.

[rampaa]

I've just tried running the repro project with .NET 8 and I can't reproduce the problem with it. So this seems to be a regression introduced in .NET 9.

[rampaa]

Shouldn't this issue be marked as regression-from-last-release? I'm not sure if that would help to raise its priority, but I figured it was worth mentioning just in case.