Add ability to stream large strings in Utf8JsonWriter/Utf8JsonReader #67337
Comments
anhadi2
added
the
api-suggestion
ghost
added
area-System.IO
untriaged
|
Tagging subscribers to this area: @dotnet/area-system-io Issue DetailsBackground and motivationI have a requirement to write large binary content in json. I have a PipeReader using which I read bytes in loop and keep appending to a list of bytes. public void WriteBinaryContent(PipeReader reader, Utf8JsonWriter writer)
{
writer.WriteStartObject();
writer.WritePropertyName("data");
byte[] byteArray = ReadBinaryData(reader).;
string base64data = Convert.ToBase64String(byteArray);
writer.WriteStringValue(base64data);
writer.WriteEndObject();
}
public byte[] ReadBinaryData(PipeReader reader)
{
List<byte> bytes = new List<byte>();
while (reader.TryRead(out ReadResult result))
{
ReadOnlySequence<byte> buffer = result.Buffer;
bytes.AddRange(buffer.ToArray());
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.Complete();
byte[] byteArray = bytes.ToArray();
return byteArray;
}The problem with this approach is excessive memory consumption. We need to keep the whole binary content in memory, convert to base64 and then write it. Instead I am proposing a way to stream large binary content. API Proposalnamespace System.Text.Json
{
public class Utf8JsonWriter : IAsyncDisposable, IDisposable
{
// This returns a stream on which binary data can be written.
// It will encode it to base64 and write to output stream.
public Stream CreateBinaryWriteStream();
}
}API Usagepublic void WriteBinaryContent(PipeReader reader, Utf8JsonWriter writer)
{
writer.WriteStartObject();
writer.WritePropertyName("data");
using (Stream binaryStream = writer.CreateBinaryWriteStream())
{
StreamBinaryData(reader, binaryStream);
binaryStream.Flush();
}
writer.WriteEndObject();
}
public void StreamBinaryData(PipeReader reader, Stream stream)
{
List<byte> bytes = new List<byte>();
while (reader.TryRead(out ReadResult result))
{
ReadOnlySequence<byte> buffer = result.Buffer;
byte[] byteArray = buffer.ToArray();
stream.Write(byteArray, 0, byteArray.Length);
stream.Flush();
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.Complete();
}Alternative DesignsNo response RisksNo response
|
|
Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis Issue DetailsBackground and motivationI have a requirement to write large binary content in json. I have a PipeReader using which I read bytes in loop and keep appending to a list of bytes. public void WriteBinaryContent(PipeReader reader, Utf8JsonWriter writer)
{
writer.WriteStartObject();
writer.WritePropertyName("data");
byte[] byteArray = ReadBinaryData(reader).;
string base64data = Convert.ToBase64String(byteArray);
writer.WriteStringValue(base64data);
writer.WriteEndObject();
}
public byte[] ReadBinaryData(PipeReader reader)
{
List<byte> bytes = new List<byte>();
while (reader.TryRead(out ReadResult result))
{
ReadOnlySequence<byte> buffer = result.Buffer;
bytes.AddRange(buffer.ToArray());
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.Complete();
byte[] byteArray = bytes.ToArray();
return byteArray;
}The problem with this approach is excessive memory consumption. We need to keep the whole binary content in memory, convert to base64 and then write it. Memory consumption is critical when using Utf8JsonWriter in the override of Instead I am proposing a way to stream large binary content. API Proposalnamespace System.Text.Json
{
public class Utf8JsonWriter : IAsyncDisposable, IDisposable
{
// This returns a stream on which binary data can be written.
// It will encode it to base64 and write to output stream.
public Stream CreateBinaryWriteStream();
}
}API Usagepublic void WriteBinaryContent(PipeReader reader, Utf8JsonWriter writer)
{
writer.WriteStartObject();
writer.WritePropertyName("data");
using (Stream binaryStream = writer.CreateBinaryWriteStream())
{
StreamBinaryData(reader, binaryStream);
binaryStream.Flush();
}
writer.WriteEndObject();
}
public void StreamBinaryData(PipeReader reader, Stream stream)
{
List<byte> bytes = new List<byte>();
while (reader.TryRead(out ReadResult result))
{
ReadOnlySequence<byte> buffer = result.Buffer;
byte[] byteArray = buffer.ToArray();
stream.Write(byteArray, 0, byteArray.Length);
stream.Flush();
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.Complete();
}Alternative DesignsNo response RisksNo response
|
|
I think it should be an |
Personally, I prefer the idea of using |
|
That's also an option, since this is the type's only purpose. |
What's the reason for this constraint? Wouldn't it be more efficient to serve it up directly instead of embedding in a JSON payload? |
We want to serve multiple binary content in the JSON with some additional fields. |
|
@anhadi2 wouldn't it make more sense to append that data after the JSON is complete? I.e.: |
This will not work for us. We want the response to be a valid JSON. |
|
Something you could do is put a URL to the data in your JSON instead of the data themselves. The data will be transmitted in binary and much more efficiently than Base64. Unless you want to persist this JSON file. |
Thanks for the suggestion. Yes, there might be other ways to do this. |
krwq
added
needs-author-action
|
This issue has been marked |
|
This issue has been automatically marked |
|
This issue will now be closed since it had been marked |
|
Such an api would be useful for us as well. We are sending data to a 3rd party api that we cannot change, and that api expects files to be sent base64 encoded as part of a json object. The documents can be up to 100 MB in size, we'd like to avoid having to load the complete document into memory. |
|
We are in the same position of sending large base64 data in JSON to a 3rd party and being unable to change the contract. @krwq You added the needs-author-action tag which seems to have led to this issue being closed. Since a couple other people have responded now requesting this feature, can it be reopened? To answer your question about #68223, it would not solve the issue that all the data needs to be present in memory at once. In the meantime I'll say to anyone else searching for a solution, if you have access to the underlying output Stream, then this ugly workaround should be viable: static void WriteStreamValue(Stream outputStream, Utf8JsonWriter writer, Stream inputStream)
{
writer.Flush();
outputStream.Write(Encoding.UTF8.GetBytes("\""));
using (CryptoStream base64Stream = new CryptoStream(outputStream, new ToBase64Transform(), CryptoStreamMode.Write, true))
{
inputStream.CopyTo(base64Stream);
}
writer.WriteRawValue("\"", skipInputValidation: true);
} |
|
Reopening. |
eiriktsarpalis
changed the title
eiriktsarpalis
changed the title
|
Back in the day, #68223 (comment) removed the public void WriteRawValue(ReadOnlySequence<char> json, bool skipInputValidation = false);overload due to the surrogate pair handling necessary for that. This new API here would require that anyways and it would also be the building block for this method's implementation. It might be a good time to reintroduce this. It would also tie in nicely with #97570 for example |
|
AI-related use case: When receiving a response from an LLM in JSON format (e.g., with response format = { "confidence": 1, "citations": [...], "answer_text": "A really long string goes here, so long that you want to see it arrive incrementally in the UI" }Currently that's not really viable to do with System.Text.Json. While we don't have an API design in mind, here are some possibilities:
Evidence requiredTBH we're still short on evidence that people really need to do this when working with an LLM, because:
Here's one bit of evidence that people will want to parse large strings in streaming JSON: https://www.boundaryml.com/blog/nextjs-rag-streaming-example. Again, it's not the only way to do this, but suggests some people will want to. If anyone reading this has clear examples of LLM-related cases where they find it desirable to process a JSON response in a streaming way, please let us know! |
|
@eiriktsarpalis I'm interested in contributing to this, our codebase still relies on less-than-ideal workarounds. I see that there was another PR that attempted to address this and was closed. I'd like to get some context on why the PR was closed to I'm well aligned with expectations. Also wanted to confirm that the scope of approved APIs only covers Utf8JsonWriter and not JsonSerializer or JsonConverters or Utf8JsonReader, is that correct? Also, the different proposed methods for |
|
@habbes can you point out the PR you're referring to? I couldn't immediately find it skimming through the issue history. |
|
@eiriktsarpalis this one: #101356 |
|
Seems like it was auto-closed due to lack of activity after receiving some comments and being marked as draft. |
dotnet-policy-service
bot
added
the
in-pr
|
@PranavSenthilnathan are you planning to do #67337 (comment) as well? |
|
Hey @davidfowl is this issue being worked on by the team? I had it on my radar to try and take a stab at it from Monday as I have some bandwidth for the next 2 weeks. But if it’s already prioritized by the .NET team I can sit this one out. |
Thanks for the interest! The APIs in this issue are implemented in these PRs already:
Yes, I have a local implementation of a byte[] converter with this new API but the perf isn't quite there yet. Once I fix that, I'll tackle string. |
|
Reopening this issue so we can track incorporation of streaming in the built-in converters as well. |
|
I've been incorporating the new APIs into our built-in byte[] and string converters, and unfortunately, I'm not seeing any performance improvements - in many cases, performance is actually worse. There are two components to consider: serialization (writing) and deserialization (reading). Serialization (Writing)On the writing side, I tested an implementation that writes in chunks when the payload is large enough. During async serialization, it would serialize and asynchronously flush chunks of the string one at a time instead of all at once. This approach didn’t show any improvement in performance benchmarks. There are a couple of reasons for this:
I ran the implementation against the TechEmpower benchmarks (and additional benchmarks in Crank), and even under high concurrency, CPU usage remained in the single digits. Basically, segmented writing didn’t improve performance because the primary bottleneck is network I/O, not serialization. Deserialization (Reading)On the reading side, the prototype reads in chunks (partially deserializing UTF-8 as it arrives) until the full JSON string is received, at which point the final C# string is constructed. This approach consistently performed worse in benchmarks. Here’s why:
RecommendationBased on these results, I recommend that we do not use the segmented APIs in our built-in converters. We should also reconsider whether the Utf8JsonWriter APIs in this proposal should still be added if we’re not going to use them ourselves. Additionally, since resumable converters aren’t public, users can’t easily create custom converters to leverage these APIs. That said, these APIs may still offer value outside of converters (like the usage example in the original API proposal). Also, in AI-related streaming scenarios (see: #67337 (comment)), they could be more applicable. Prototype: #112129 Benchmark diffs
|
Where is the memory use in the benchmarks? Do you have a GC trace we can look at? I wouldn't expect more throughput, I'd expect better scalability and memory usage if you were doing LOTS of these in concurrently. It might also be worth running an end-to-end ASP.NET Core scenario with a large JSON payload. |
|
Because of pooling, the memory numbers in the microbenchmarks are not useful (the warmup iterations populate the pool and the test iterations just use memory from the populated pool). However, I did run the Crank 400K JSON benchmark for a more representative workload. Here is a diff:
Note that this does show GC improvements in some cases (LOH/POH size) and regressions in others (Gen 1/Gen 2 size), but overall the CPU usage is the same. If you would like to see a different ASP.NET-specific benchmarks feel free to point me to them. |
Uh oh!
There was an error while loading. Please reload this page.
EDIT See #67337 (comment) for an API proposal.
Background and motivation
I have a requirement to write large binary content in json.
In order to do this, I need to encode it to base64 before writing.
The resultant json looks like this:
I have a PipeReader using which I read bytes in loop and keep appending to a list of bytes.
I then convert the list into a byte array, then convert it to base64 string and use
WriteStringValueto write it.The problem with this approach is excessive memory consumption. We need to keep the whole binary content in memory, convert to base64 and then write it.
Memory consumption is critical when using Utf8JsonWriter in the override of
JsonConverter.Write()method in a web application.Instead I am proposing a way to stream large binary content.
API Proposal
API Usage
Alternative Designs
No response
Risks
No response
The text was updated successfully, but these errors were encountered: