Reading partial JSON responses as they're streamed

[JamesNK]

I want to combine a JSON schema structured response and streaming. However, that causes the response text to be an incomplete JSON string until the streaming response is finished.

Incomplete JSON is challenging to read with System.Text.Json. All the high-level APIs expect JSON to be complete: JsonDocument.Parse(json), JsonSerializer.Deserialize(json). I don't want to wait until the JSON is complete. The reason I have streaming enabled is to show results to the user as they become available.

I've been thinking about reading partial JSON using Utf8JsonReader: https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/use-utf8jsonreader#read-from-a-stream-using-utf8jsonreader

I'd use Utf8JsonReader to read the content into my POCO manually, and then stop when the reader throws an error. That way I at least populate the properties that are available. Utf8JsonReader is a pain to use, but it's not an insurmountable problem.

However, even when doing this I don't think I can get the values of strings that are inprogress. For example, the following JSON that's inprogress:

{ "message": "this is the long message but it's not completed yet

I don't think I would be able to access the message value until it has closing double quotes:

{ "message": "this is the long message but it's not completed yet"

Maybe the solution here is to ask AI to format the output so it splits the message up into an array of strings:

{ "message": ["this is the long message ", "but it's not completed yet

Is that good thing to do? I don't know. I'd need control over the response schema which I do here in this situation, but it might not always be true.

It feels like JSON + streaming is a common problem, with a reasonably hard solution: use Utf8JsonReader. APIs in MEAI to make this easier, and/or samples and guidance would be useful.

[stephentoub]

c: @eiriktsarpalis

[eiriktsarpalis]

Incomplete JSON is challenging to read with System.Text.Json. All the high-level APIs expect JSON to be complete: JsonDocument.Parse(json), JsonSerializer.Deserialize(json). I don't want to wait until the JSON is complete. The reason I have streaming enabled is to show results to the user as they become available.

The only STJ method that really supports streaming is DeserializeAsyncEnumerable, which requires that the payload is top-level JSON values. What would a general-purpose partial/streaming deserialization method look like? I struggle to think of a good way to meaningfully achieve that. If we had something like

public static bool DeserializePartial<T>(ReadOnlySpan<byte> partialJson, ref T? result, DeserializationState? state = null);

How would the consumer of result know what bits of it are new?

I don't think I would be able to access the message value until it has closing double quotes:

It's something we're working at improving for .NET 10 (cf. dotnet/runtime#67337) but this would only work at the reader layer; fundamentally it won't change how values are being returned at the deserialization layer.

{ "message": ["this is the long message ", "but it's not completed yet

Is that good thing to do? I don't know. I'd need control over the response schema which I do here in this situation, but it might not always be true.

How would this work assuming that there are trailing properties after the long message that the deserializer would also need to bind to the POCO that is being returned?

[JamesNK]

I'm not trying to do this anymore.